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systems,  the  AN/APN-147  and  AN/ASN-35.  The  effectiveness  of  the  technical  data  and  the  existing  TOs  for  the 
systems  were  then  evaluated  by  determining  which  provided  the  best  support  for  technicians  troubleshooting  the 
systems.  This  was  accomplished  by  inserting  faults  into  the  equipment  and  having  a technician  troubleshoot  the 
system  using  one  of  the  three  types  of  data.  Job  performance  tests  were  used  to  measure  the  performance  of  the 
technicians.  Technicians  with  three  levels  of  experience  participated  in  the  study.  They  were:  apprentice  teduiidans 
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experienced  technicians  with  more  than  six  months  experience  on  the  systems.  The  results  indicate  that:  (1) 
apprentice  technicians  are  able  to  troubleshoot  more  effectively  when  using  FPTAs  than  when  using  LTTAs;  (2) 
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apprentice  technicians  using  FPTAs  are  able  to  troubleshoot  as  effectively  is  experienced  technicians  using  TOs,  and 
(4)  fewer  parts  are  replaced  unnecessarily  when  FPTAs  or  LTTAs  are  used  than  when  TOs  ate  used. 


SUMMARY 


PURPOSE 

The  purpose  of  the  project  described  in  this  report  was  to  conduct 
an  evaluation  comparing  fully  proceduralized  troubleshooting  aids 
(FPTAs),  logic  tree  troubleshooting  aids  (LTTAs),  and  technical  orders 
(TOs)  in  terms  of: 

1.  The  costs  involved  in  developing  each  type  of  aid. 

2.  The  technical  accuracy  of  each  type  of  data. 

3.  The  effectiveness  of  each  type  of  data  for  supporting 
maintenance  personnel  of  varying  experience  levels  in 
performing  troubleshooting  tasks. 


METHOD 

Cost  comparison:  The  available  data  on  costs  to  produce  each  of 

the  three  types  of  technical  data  were  obtained  from  FPTA  and  LTTA 
contractors,  and  from  the  Air  Force  Logistic  Centers  responsible  for  TO 
procurement.  Due  to  differences  in  coverage,  developer  experience,  data 
base  and  types  of  systems  for  which  the  technical  data  were  developed, 
difficulty  was  experienced  in  arriving  at  a single  measure  which  would 
permit  a meaningful  comparison  of  the  three  document  types.  The  approach 
selected  was  to  base  the  analysis  on  the  cost  to  produce  a page  unit.  A 
page  unit  was  defined  as  a standard  8 1/2-  by  11- inch  format.  Use  of 
this  standard  made  it  necessary  to  convert  the  A-  by  8-lnch  page  format 
used  in  the  LTTA  checkout  procedures  for  organizational  level  maintenance 
into  the  standard  page  unit.  No  other  page  conversions  were  required. 

Technical  accuracy:  Review  of  worldwide  Air  Force  maintenance 

actions  was  undertaken  to  provide  identification  of  maintenance  areas 
which  could  serve  as  representative  candidate^  for  development  of 
troubleshooting  test  problems.  Thirty  specific  troubleshoot inf  test 
problems  were  selected  and  distributed  throughout  the  AN/APN-147  (radar) 
and  AN/ASN-35  (computer)  systems  of  the  C-141A  aircraft.  Each  problem 
was  inserted  into  working  system  equipment.  The  effects  on  system 
operation  were  Identified  and  a systematic  evaluation  of  the  accuracy  of 
the  technical  documentation  types  in  supporting  the  isolation  of  the 
faults  was  undertaken.  Records  were  maintained  on  the  errors  found  in 
the  data  and  on  whether  the  procedure  led  to  isolation  of  the  problem. 
Those  faults  which  could  be  successfully  isolated  by  each  of  the  three 
types  of  technical  data  were  retained  for  consideration  as  a test 
problem  in  the  field  data  collection  phase. 

Experimental  evaluation  of  effectiveness  of  each  technical  data 
type:  Three  experience  levels  of  Air  Force  technicians  in  the  Avionics 

Inertial  and  Radar  Navigational  Systems  Specialty  (AFSC  328X4)  served 


as  subjects.  The  experience  levels  were:  recent  graduates  of  the 

technical  training  course  (no  field  experience);  technicians  with  six 
months  or  less  experience;  and  technicians  with  more  than  six  months 
experience.  Eighteen  technicians  at  each  experience  level  were  used  as 
subjects.  A determination  was  made  that  it  would  not  be  appropriate  to 
test  the  no-experience  group  on  technical  orders.  This  decision  made  it 
necessary  to  use  two  different,  but  similar,  experimental  designs.  Both 
designs  involved  repeated  measures  on  subjects.  In  the  first  design, 
the  no-experience  group  was  tested  using  two  types  of  data,  FPTAs  and 
LTTAs.  In  the  second  design,  the  two  experienced  groups  of  subjects 
were  tested  using  all  three  types  of  data.  Data  collection  was  accom- 
plished at  Keesler  Technical  Training  Center  and  at  three  Military 
Airlift  Command  bases:  McChord,  Travis,  and  McGuire. 


RESULTS 

The  cost  comparison,  primarily  due  to  the  fact  that  sufficient 
comparable  data  could  not  be  assembled,  produced  inconclusive  results. 
However,  the  available  data  did  suggest  that  the  costs  of  developing 
fully  procedurallzed  troubleshooting  aids  in  a newly  developed  system 
where  all  support  data  would  also  have  to  be  generated  may  not  be  much 
different  from  the  costs  incurred  in  technical  order  development. 

Errors  in  technical  accuracy  were  found  in  both  the  FPTA  and  LTTA. 

The  errors  discovered  vary  in  their  Impact  on  performance  from  negligible 
to  completely  disruptive.  However,  no  accuracy  errors  were  found  that  were 
considered  beyond  solution.  Increased  emphasis  on  quality  control  would 
eliminate  most  of  the  accuracy  errors  found. 

The  results  of  the  experimental  evaluation  clearly  demonstrate  that 
the  use  of  procedurallzed  troubleshooting  aids  resulted  in  significantly 
better  troubleshooting  than  the  use  of  the  TO.  This  finding  held  for 
two  of  the  three  measures:  proportion  of  problems  solved  and  spare 

parts  consumed.  For  the  third  measure,  time  to  troubleshoot,  the  use  of 
TOs  resulted  in  better  performance  at  the  organizational  level  but  not 
at  the  intermediate  level  of  maintenance. 

In  comparing  the  FPTA  against  the  LTTA,  the  FPTA  was  superior  far 
the  no-experience  group  on  all  measures.  For  experienced  technicians, 
use  of  the  FPTA  resulted  in  slightly  better  performance  on  all  measures 
except  time  to  troubleshoot  at  the  organizational  level. 

In  addition  to  the  objective  performance  date,  subjective  opinion 
data  of  the  technicians  provided  unqualified  support  for  the  use  of  the 
FPTA  as  the  most  favored  of  the  three  types  of  technical  documentation. 
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EVALUATION  OF  THREE  TYPES  OF  TECHNICAL  DATA  FOR  TROUBLESHOOTING: 
RESULTS  AND  PROJECT  SUMMARY 

Section  I 

INTRODUCTION 


The  steadily  Increasing  costs  of  maintaining  Air  Force  operational 
systems,  combined  with  current  austerity  programs,  make  it  essential 
that  new  ways  of  reducing  maintenance  costs  be  identified  and  implemented. 
One  way  of  reducing  costs  is  to  improve  the  effectiveness  of  maintenance 
personnel,  and  thereby  reduce  manpower  requirements.  Manpower  costs  for 
maintenance  personnel  presently  account  for  15  percent  of  the  Air  Force 
budget.  Thus,  the  area  of  maintenance  manpower  costs  offers  the  potential 
for  significant  cost  savings. 

One  approach  which  has  significant  potential  for  Improving  the 
efficiency  of  maintenance  personnel  and  reducing  costs  is  the  development 
of  improved  types  of  technical  data  for  use  by  maintenance  personnel. 

Many  types  of  improved  technical  data  have  been  proposed  for  use  by 
Air  Force  and  other  Department  of  Defense  agencies.  These  include  data 
such  as: 


o Maintenance  Dependency  Charts  and  Schematic  Block  Diagrams 
(Mil-M- 38799) 

o Functionally  Oriented  Maintenance  Manuals  (MIL-M-24100B) 

o Logic  Tree  Troubleshooting  Aids  (MIL-M-38800A) 

o Fully  proceduralized  Troubleshooting  Aids  (AFHRL-TR-73-43(I)) 

The  new  types  of  data  represent  attempts  to  improve  the  content 
and  method  of  presenting  Information  and  directions  for  technicians  to 
use.  Proponents  of  the  new  types  of  data  usually  claim  that  this  type 
of  data  has  significant  advantages  over  standard  technical  orders  and 
other  types  of  Improved  technical  data.  However,  there  is  usually 
little  empirical  data  to  support  these  claims.  The  potential  financial 
impact  of  implementing  new  types  of  technical  data  makes  it  essential 
that  decision  makers  have  valid  information  available  on  the  benefits 
of  each. 

The  project  reported  in  this  technical  report  was  designed  to 
provide  empirical  data  on  the  relative  effectiveness  for  supporting 
troubleshooting  activities  of  two  of  the  new  types  of  data  in  comparison 
to  standard  technical  orders  (TOs) . The  two  new  types  of  data  are 
Fully  Proceduralized  Troubleshooting  Aids  (FPTAs)  and  Logic  Tree 


13 


Troubleshooting  Aids  (LTTAs) . Both  FPTAs  and  LTTAs  provide  step-by-step 
procedures  for  troubleshooting.  They  differ  in  the  format  in  which  the 
procedures  are  presented,  the  level  of  detail,  and  the  procedures  used 
to  develop  the  data. 

Proponents  of  both  FPTAs  and  LTTAs  claim  that  their  type  of  data  has 
significant  advantages  over  standard  technical  orders.  They  claim  that 
the  use  of  FPTAs  and  LTTAs  will  result  in; 

1.  More  efficient  maintenance  by: 

a.  Reducing  the  time  required  for  troubleshooting. 

b.  Reducing  utilization  of  spare  parts  (due  to  fewer  removals 
of  good  parts) . 

2.  More  efficient  use  of  personnel  by: 

a.  Permitting  less  experienced  technicians  to  troubleshoot 
effectively  (making  it  possible  for  them  to  be  productive 
earlier  in  their  enlistments). 

b.  Facilitating  the  transfer  of  personnel  experienced  on  one 
system  to  an  unfamiliar  system. 

c.  Reducing  training  requirements. 

ProP°nents  of  FPTAs  believe  that  the  more  thorough  analysis  used  in 
the  development  of  FPTAs  produces  data  of  a much  better  quality. 
Consequently,  they  believe  that  FPTAs  will  achieve  all  of  the  above 
advantages  with  a greater  degree  of  success.  They  also  believe  that 
the  rigidly  specified  FPTA  development,  validation,  and  verification 
procedures  will  produce  data  that  are  technically  more  accurate  (have 
fewer  errors)  and  more  complete.1 

Proponents  of  LTTAs  believe  that  this  type  of  data  can  support 
troubleshooting  tasks  as  effectively  as  FPTAs  at  much  less  cost.  They 
believe  that  the  rigid  developsient  procedures  specified  for  FPTA 
development  are  not  necessary  to  produce  high  quality  data  and  that 
s gfllf leant  economics  can  be  obtained  by  allowing  the  contractor  to  have 
more  freedom  in  the  development  process.  They  also  believe  that  the  high 
evel  of  detail  provided  In  the  FPTA  Is  not  necessary  except  for  very 
inexperienced  personnel.  They  do  not  feel  that  the  additional  cost 
incurred  in  providing  the  detail  (in  terms  of  development  cost  and  extra 
pages  to  produce  and  maintain)  are  justified  for  the  relatively  few 
technicians  who  require  this  level  of  detail. 


The  FPTA  technology  was  developed  by  the  Air  Force  Human  Resources 
Laboratory  as  a result  of  s 15  year  program  to  improve  technical  data 
for  maintenance.  See  Foley  (1973)  for  a description  of  the  research 
program  end  a more  complete  deecription  of  the  benefits. 


As  indicated  above,  the  use  of  proceduralized  data  for  trouble- 
shooting could  have  a significant  impact  on  the  cost  of  Air  Force 
maintenance.  Theoretically,  a proceduralized  approach  permits  fault 
isolation  in  a shorter  period  of  time  in  that  it  eliminates  false 
starts  and  blind  alleys  in  the  troubleshooting  process.  This  saving  in 
troubleshooting  and  repair  time  can  result  in  two  separate  benefits  to 
the  Air  Force,  more  productive  utilization  of  the  maintenance  technician, 
and  more  in-commission  (or  on-line)  time  for  Air  Force  equipment  and 
systems.  An  additional  benefit  foreseen  from  the  increase  in  trouble- 
shooting efficiency  is  that  the  Air  Force  can  reduce  its  spare  parts 
stock  inventory  requirements.  The  present  trial-and-error  method  of 
troubleshooting  (question,  remove,  replace,  and  test)  contributes 
in  many  instances  to  unnecessary  replacement  of  good  parts  and  components. 
The  number  of  parts  unnecessarily  used  in  repair  activities  can  be 
significantly  reduced  and  possibly  largely  eradicated  through  procedural- 
ized approaches  to  troubleshooting. 

The  use  of  proceduralized  troubleshooting  has  significant  promise 
for  improving  the  efficient  use  of  maintenance  personnel.  This  is  due 
primarily  to  the  fact  that  extensive  knowledge  of  the  system  is  not 
needed  to  effectively  troubleshoot.  Theoretically,  then,  it  should  be 
possible  for  a technician  with  limited  experience  to  troubleshoot 
effectively  using  proceduralized  data.  Also,  it  should  be  possible 
for  a technician  with  experience  on  one  system  to  troubleshoot  a 
similar  but  unfamiliar  system.  The  implications  for  savings  in  both 
training  costs,  and  in  manpower  costs  through  longer  productive  utili- 
zation of  the  airman  in  his  initial  enlistment  period,  and  in  the 
transition  of  experienced  technicians  between  systems  are  highly 
significant. 

The  potential  impact  of  proceduralized  troubleshooting  approaches 
on  training  deserves  special  comment.  In  the  present  technical  train- 
ing concept,  large  blocks  of  training  time  must  be  devoted  to  the 
presentation  of  theory.  Frequently,  the  theory  is  presented  at  the 
expense  of  direct  hands-on  practice.  Thus,  the  technician  arriving  at 
an  operational  assignment  from  technical  training  is  not  capable  of 
assuming  maintenance  duties.  He  must  be  placed  in  additional  training 
in  his  unit  of  assignment  to  obtain  the  practical  experience  required 
to  permit  him  to  work  with  the  equipment.  The  training  used  with 
proceduralized  technical  documentation,  as  a rule,  has  emphasized 
hands-on  practice  to  obtain  equipment  familiarity.  Thus,  an  airman 
trained  in  such  a manner  is  able  to  function  in  a productive  capacity 
much  earlier  in  his  term  of  enlistment  than  is  true  with  present 
training  methods.  With  proceduralized  documentation,  the  technician 
does  not  require  a knowledge  of  the  theory  underlying  the  operating 
system  to  maintain  it,  and  this  elimination  represents  a savings  in 
both  training  time  and  costs.  Each  technician  has  a longer  productive 
period  in  his  first  enlistment.  Further,  this  approach  permits  a more 
intensive  selection  process  for  additional  training  to  take  place; 
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thus,  permitting  the  Air  Force,  to  a greater  extent  than  is  presently 
true,  to  invest  its  training  monies  on  those  technicians  showing  greatest 

promise  for  return  of  investment. 

The  present  project  was  designed  to  collect  data  relative  to 
ability  of  FPTAs  and  LTTAs  to  improve  maintenance  efficiency  and  their 
effectiveness  for  improving  the  performance  of  technicians  with  varying 
levels  of  experience.  The  impact  of  FPTAs  and  LTTAs  on  training 
requirements  was  not  directly  assessed  in  this  project.  However,  it 
was  possible  to  obtain  a limited  amount  of  data  relative  to  the  potential 
of  the  FPTAs  and  LTTAs  for  facilitating  transfer  of  technicians  to  new 
systems.  These  data  are  summarized  and  discussed  separately  in  Appendix  C. 


STATEMENT  OF  THE  PROBLEM 

The  purpose  of  the  project  described  in  this  report  was  to  conduct 
an  evaluation  comparing  fully  proceduralized  troubleshooting  aids,  logic 
tree  troubleshooting  aids,  and  technical  orders  in  terms  of: 

1.  The  cost  to  develop  each  type  of  data. 

2.  The  technical  accuracy  of  each  type  of  data. 

3.  The  effectiveness  of  each  type  of  data  for  supporting 
maintenance  personnel  of  varying  levels  performing  trouble- 
shooting tasks. 

Effectiveness  of  each  type  of  technical  data  was  evaluated  through 
accomplishment  of  a research  design  which  permitted  the  following 
questions  to  be  addressed: 

1.  Does  quality  of  performance  with  the  three  types  of  documenta- 
tion depend  upon  experience  level  of  the  technician? 

2.  Does  FPTA  performance  differ  from  LTTA  performance?  Are 
both  FPTA  and  LTTA  performance  different  from  TO  performance? 

3.  Is  there  an  interaction  between  type  of  technical  data  and 
level  of  experience?  If  so,  what  is  the  nature  of  the 
interaction? 

4.  Does  performance  with  the  three  types  of  technical  documenta- 
tion differ  for  organizational  and  intermediate  level 
maintenance  troubleshooting? 

5.  Is  there  a statistical  interaction  between  level  of  maintenance 
troubleshooting  and  level  of  experience?  If  so,  what  is  the 
nature  of  the  interaction? 
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6.  Is  there  a statistical  interaction  between  FPTA  or  LTTA 
technical  data  and  level  of  maintenance  troubleshooting?  If 
so  what  is  the  nature  of  the  interaction? 

7.  Is  there  an  interaction  among  types  of  technical  documentation, 
experience  level  and  level  of  maintenance  troubleshooting?  If 
so,  what  is  the  nature  of  the  interaction? 


GENERAL  APPROACH 

To  evaluate  the  three  types  of  data,  it  was  necessary  to  first 
develop  FPTAs  and  LTTAs  for  a test  system.  FPTAs  and  LTTAs  were 
developed  under  contract  for  the  AN/APN-147  doppler  radar  and  AN/ASN-35 
computer  as  configured  in  the  C-141.  FPTAs  and  LTTAs  were  developed  to 
provide  procedures  for  all  routine  troubleshooting  tasks  performed  at 
the  organizational  and  intermediate  levels  of  maintenance.  The  develop- 
ment of  the  aids  and  reasons  for  selecting  the  test  systems  are  described 
in  Section  II  of  this  volume. 

An  analysis  of  the  comparative  costs  of  developing  the  three  types 
of  data  was  made  by  collecting  data  on  the  costs  of  preparing  the  FPTAs 
and  LTTAs  developed  in  this  study.  The  results  of  this  analysis  are 
summarized  in  Section  IV. 

The  analysis  of  the  accuracy  of  the  FPTAs  and  LTTAs  was  made  by 
selecting  representative  faults  which  occur  in  the  systems  and  using 
each  type  of  data  to  isolate  the  fault.  The  analysis  considered  whether 
the  data  did  in  fact  lead  to  successful  isolation  of  the  problem  and  the 
number,  type,  and  seriousness  of  errors  identified  in  the  data.  The 
results  of  this  analysis  are  given  in  Section  II  of  this  volume  of  the 
report. 

The  major  emphasis  of  the  study  was  directed  toward  determining  the 
effectiveness  of  each  type  of  data  for  supporting  the  technician 
performing  troubleshooting  tasks.  There  are  several  approaches  which 
may  be  taken  in  determining  the  effectiveness  of  technical  data  for 
supporting  troubleshooting  tasks.  The  most  valid  technique  would  be 
to  introduce  the  data  into  the  operational  environment  for  use  in 
regular  troubleshooting  tasks  in  support  of  the  unit's  mission. 

However,  this  approach  usually  is  not  feasible  since  the  adequacy  of 
the  data  is  unproven  and  its  use  may  interfere  with  the  mission  of  the 
test  unit.  This  approach  is  also  handicapped  by  the  fact  that  it  is 
very  difficult  to  obtain  adequate  measures  of  maintenance  effectiveness 
in  an  operational  environment.  Due  to  the  interference  of  many 
extraneous  factors,  it  is  not  possible  to  obtain  accurate  data  on 
important  factors  such  as  rate  of  successful  completion  of  the  trouble- 
shooting task,  time  to  complete  the  task,  and  errors  made.  As  a result, 
it  is  usually  necessary  to  rely  on  the  opinions  of  or  ratings  by 
technicians,  supervisors,  and  other  observers  as  measures  of 


effectiveness  in  the  operational  setting.  This  type  of  data  is 
extremely  subject  to  bias  and  easily  influenced  by  outside  factors. 
Therefore,  this  approach  was  considered  unacceptable  for  this  project 
and  was  not  used. 

A second  approach  is  to  simulate  the  maintenance  environment  by 
having  technicians  perform  maintenance  tasks  using  each  type  of  data 
under  controlled  conditions.  Performance  tests  are  used  to  measure  the 
technicians'  tasks.  For  each  test,  the  technician  is  told  to  trouble- 
shoot the  system  or  component  of  the  system  using  the  data  being 
evaluated.  A trained  observer  monitors  his  performance  and  records 
errors  made,  problems  encountered  in  using  the  data,  whether  the  task  is 
correctly  accomplished,  and  the  time  required  to  complete  the  task.  This 
technique  closely  simulates  the  maintenance  environment  while  controlling 
most  of  the  extraneous  factors  which  affect  performance.  However,  the 
technique  has  several  disadvantages.  It  requires  that  actual  prime 
equipment  be  made  available  and  dedicated  to  the  project.  For  an 
operational  system,  such  equipment  is  often  very  difficult  to  obtain.  It 
requires  the  time  of  several  technicians  to  complete  a number  of 
maintenance  tasks.  Thus,  a significant  number  of  man-hours  are  required 
to  obtain  an  adequate  sample  of  performance.  Finally,  a trained  observer 
is  required  to  monitor  the  performance  of  the  technician.  In  most  cases, 
monitoring  must  be  on  a one-to-one  basis.  Consequently,  a significant 
amount  of  observer  time  is  required.  In  summary,  this  approach  should 
provide  the  most  valid  data  but  it  is  very  time  consuming  and  costly. 

A third  approach  which  has  been  used  in  some  studies  is  to  attempt 
to  simulate  the  maintenance  task  through  use  of  paper-and-pencil  tests. 

In  this  approach,  the  technician  is  given  the  technical  data  being 
evaluated  and  information  on  the  results  that  would  be  obtained  at  each 
test  point  if  the  prescribed  test  were  actually  performed.  The  techni- 
cian uses  the  technical  data  to  determine  what  checks  should  be  made 
and  refers  to  the  information  package  for  the  test  results*.  The 
technician's  performance  is  evaluated  on  whether  he  successfully 
identifies  the  fault.  This  approach  has  the  advantages  of  not  requir- 
ing the  prime  equipment,  requiring  a relatively  small  amount  of  the 
technicians '8  time,  and  being  suitable  for  group  administration. 

However,  this  approach  has  the  very  significant  disadvantage  of  not 
adequately  simulating  the  maintenance  environment.  There  are  several 
factors  which  influence  whether  a technician  is  able  to  use  a manual 
to  troubleshoot  besides  being  able  to  follow  a sequence  of  checks.  For 
example,  does  the  technical  data  clearly  tell  the  technician  where  a 
test  point  is  located?  If  he  cannot  find  the  test  point,  he  cannot  make 
the  check.  Thus,  it  is  very  possible  that  a technician  may  be  able  to 
do  well  on  a paper-and-pencil  test  and  still  not  be  able  to  solve  the 
problem  using  the  test  data  on  the  actual  equipment.  Thus,  the  validity 
of  this  approach  is  questionable. 
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Although  the  performance  test  approach  is  significantly  more 
expensive  and  time  consuming,  it  was  believed  that  the  validity  and 
other  benefits  of  the  approach  justified  the  cost  and  time.  This  approach 
was  used.  The  procedures  and  results  are  described  in  Sections  V and  VI. 

A detailed  description  of  the  experimental  procedures  utilized  is 
presented  in  AFHRL-TR-76-74 (II) . 


TEST  FACILITIES 

As  indicated  above,  the  performance  test  approach  requires  dedica- 
ted access  to  the  prime  equipment  for  extensive  periods  of  time.  It 
was  realized  that  it  would  not  be  possible  to  obtain  sufficient  access 
to  test  benches  at  operational  bases  without  causing  an  unacceptable 
degree  of  interference  with  their  mission.  Further  mission  requirements 
usually  create  an  unacceptable  degree  of  interference  with  the  test 
schedule.  To  ensure  adequate  access  to  the  equipment  at  all  test 
locations,  a 40-foot  Air  Force  van  was  equipped  with  bench  mockups  of 
the  AN/APN-147  and  AN/ASN-35  to  provide  a transportable  test  facility. 
It  was  also  recognized  that  it  would  be  extremely  difficult  to  obtain 
adequate  access  to  C-141  aircraft  for  use  with  the  organizational  level 
tests.  Thus,  a mockup  of  the  C-141  cockpit  and  the  avionics  equipment 
bays  of  the  C-141  ms  developed  and  installed  in  the  test  van.  The 
cockpit  mockup  provided  a means  of  simulating  the  flightline  trouble- 
shooting tasks  wihtout  requiring  access  to  an  aircraft.  It  consists  of 
mockups  of  all  instrument  panels  in  the  cockpit.  All  AN/APN-147  and 
AN/ASN-35  components  were  "live"  and  functioned  in  the  same  manner  as 
the  actual  cockpit.  All  other  components  on  the  instrument  panels  were 
represented  by  photographs.  Figure  1 is  a photograph  of  the  exterior 
of  the  van.  Figure  2 is  a view  of  the  cockpit  mockup  area  inside  the 
van.  Figures  3 and  4 are  views  of  the  bench  mockups  contained  in  the 
van. 


CONTENTS  OF  THIS  REPORT 

The  report  is  complete  in  three  volumes.  This  volume  (AFHRL-TR- 
76-74  (I))  provides  a summary  of  the  entire  project.  The  remaining 
volumes  provide  detailed  information  on  various  portions  of  the  study 
for  those  requiring  additional  detailed  information  for  analysis  or 
replication.  The  two  additional  volumes  are: 

Volume  II,  Treatment  Plan.  This  volume  provides  a detailed 
description  of  the  experimental  procedures  used  in  the 
experimental  evaluation  portion  of  the  project. 

Volume  III,  Test  Administrator's  Guide.  This  volume  provides 
instructions  and  guidance  for  administering  the  job  performance 
tests  used  to  measure  performance  of  technicians  using  FPTAs, 
LTTAs,  and  TOs. 


Figure  1.  Exterior  View  of  Air  Force  Van  I'sed  to  House  the  C-141A  Cockpit  Mockup  and 
the  Radar  and  Computer  Maintenance  Benches  used  in  Field  Data  Collection 


Kigure  3.  View  of  the  Radar  Maintenance  Bench 


Section  II 


SELECTION  OF  TEST  SYSTEM  AND  DEVELOPMENT  OF  DATA 


SELECTION  OF  THE  TEST  SYSTEM 

As  indicated  in  Section  I,  two  electronic  subsystems  of  the  C-141 
aircraft  were  selected  as  the  test  bed.  These  are  the  AN/APN-147 
doppler  radar  and  its  associated  computer,  the  AN/ASN-35.  These  systems 
were  selected  for  a number  of  reasons  including: 

1.  The  systems  are  moderately  complex.  They  are  complex  enough 
to  test  the  technology  and  small  enough  to  permit  development 
of  FPTAs  and  LTTAs  at  a reasonable  cost. 

2.  The  systems  are  taught  as  typical  sets  in  the  ATC  Course 
3ABR32834  (Inertial  and  Radar  Navigation  System  Specialist). 

3.  The  technical  orders  for  the  AN/APN-147  had  recently  been 
rewritten.  Thus,  these  TOs  were  representative  of  current 
T Os  and  provided  a good  basis  for  comparison  with  the  FPTAs 
and  LTTAs. 

4.  Some  AN/APN-147  and  AN/ASN-35  equipment  was  already  available 
to  AFHRL. 


DESCRIPTION  OF  THE  THREE  TYPES  OF  DATA 

There  are  significant  differences  in  the  presentation  of  information 
for  the  three  types  of  data.  These  differences  primarily  involve  the 
extent  to  which  the  data  are  arranged  for  convenient  use,  the  level  of 
detail,  and  the  degree  to  which  instructions  are  proceduralized.  The 
contents  and  presentation  of  each  type  of  data  are  described  in  the 
following  paragraphs. 

Conventional  troubleshooting  manuals  (TOs)  are  the  least  procedural- 
ized of  the  three  types  of  data.  Most,  if  not  all,  of  the  information 
required  to  troubleshoot  the  equipment  is  contained  within  the  manual. 

The  technician  must  decide  what  data  he  will  use  and  how  and  when  he 
will  use  it.  The  manual  generally  contains  the  following  types  of 
information. 

1.  A list  of  test  equipment  and  special  tools  placed  in  the  front 
of  manual,  applicable  to  all  sections. 
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2.  Major  sections  covering  equipment  units  of  the  system.  Each  of 

the  sections  contains  several  different  types  of  illustrations 

such  as: 

a.  photographs  or  line  drawings  identifying  modules  and  major 
and  minor  test  points. 

b.  a troubleshooting  chart  containing  a checkout  procedure 
(somewhat  proceduralized)  and  a symptom-cause  table  that 
indicates  probable  malfunctions  when  a checkout  step  is 
failed. 

c.  voltage  and  resistance  diagrams. 

d.  voltage  waveforms. 

e.  block  diagrams. 

f.  schematic  diagrams. 

g.  wiring  diagrams. 

3.  A functional  description  of  the  equipment  and  detailed  circuit 

description. 

The  troubleshooting  procedure  will  generally  isolate  the  malfunction 
to  a particular  stage  and  then  refer  the  technician  to  the  appropriate 
diagram  or  illustration.  The  manuals  are  prepared  assuming  that  the 
technician  has  experience  and  training  in  the  use  of  test  equipment,  in 
locating  most  parts  within  the  equipment,  and  in  interpreting  schematic 
diagrams.  Figures  5 and  6 contain  samples  of  TO  pages.  Figure  5 is  a 
page  from  the  checkout  section  of  the  TO;  Figure  6 contains  a page  from 
a symptom-cause  table. 

The  logic  tree  troubleshooting  aids  are  somewhat  more  proceduralized 
than  conventional  manuals  in  that  specific  instructions  are  provided  both 
in  the  checkout  procedure  and  in  the  troubleshooting  procedure.  The 
troubleshooting  procedures  begin  at  the  step  in  the  checkout  procedure 
that  produced  the  failure.  Figure  7 contains  a checkout  procedure  with 
a selected  trouble  outlined.  LTTAs  contain  a cross-reference  from 
failed  checkout  procedure  to  troubleshooting  procedure  (Figure  8).  This 
index  may  either  reference  the  proper  troubleshooting  tree  or  pinpoint 
a fault  directly.  The  troubleshooting  logic  trees  (Figure  9)  give  exact 
procedures  to  be  performed  and  ask  questions  about  results  obtained 
while  monitoring  voltage,  waveforms,  and/or  resistances  at  specific  points 
in  the  suspect  circuit.  The  yes  or  no  answer  to  these  questions  provides 
the  logic  in  the  fault  isolation  procedure.  Each  path  ends  with 
instructions  to  replace  a faulty  component.  Sometimes  a path  will  first 
identify  the  faulty  stage  or  assembly  and  will  direct  the  technician  to 
perform  the  checkout  procedure  for  that  stage  of  assembly. 
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T O.  5N1-3-8-2 


6-8.  Set  test  harness  controls  as  follows: 

a HEADING  TRACK  SIMULATOR 
switch  to  1NT. 

b.  Deleted 


c.  Deleted 


d.  Set  audio  oscillator  to  12,240  i 10 
Hr  at  10V  P/P  (3VR.MS). 

e.  26VAC  EXC  switch  to  1NT. 

f.  1.ANO/SK  \/SS  switch  to  SKA. 

g.  DRIFT  ANGLE  switch  to  OUT. 

h.  HEADING  TRACK  SIMULATOR  to 
000.0  degrees. 

1.  COMPUTER  REFERENCE  TRACK 
switch  to  IN. 

6-9  Set  control  indicator  (C-3748A.  etc) 
as  follows: 

a.  OFF/VIAN/AUTO  switch  to  OFF. 

b.  DISTANCE  TO  GO,  STAGE  I to 
100  NM. 


c.  DISTANCE  CROSS  TRACK  to 
00.0NM  RIGHT. 

d.  DESIRED  TRACK  ANGLE,  STAGE 
I to  000.0*. 

e.  Stage  switch  to  STAGE  1 ACTIVE. 

6-10.  Set  control  indicator  auxiliary 
(C3749A,  etc.)  as  follows: 

a.  NAV/DHOP  switch  to  N A \ . 

b.  DISTANCE  CROSS  TRACK  to 
00 . ON M RIGHT. 

c.  Selector  switch  to  AUXILIARY 
STAGE  I,  MAIN  CROSS  TRACK  STAGE  II. 

6-11.  Turn  power  on  and  allow  time  for 
system  and  test  equipment  warmup. 

6-2  Change  14 


6-12.  TEST  PROCEDURE. 

6-13.  Set  control  indicator  OFF/MAN/ 
AUTO  switch  to  MAN. 

a.  DISTANCE  TO  GO  counter  should 
decrease. 

b.  DISTANCE  CROSS  TRACK  counter 
should  not  change. 

c.  Indicator- Multiple  (1D939A)  OFF 
flag  is  not  in  view. 

d.  Indicator  pointer  is  at  0 . 

e.  Indicator  cross  track  bar  is  •>> 
center. 

f.  Indicator  DISTANCE  TOGOcou.itei 
is  decreasing  and  reads  the  game  as  DIS- 
TANCE TOGO  counter  on  control  u.die.A 

6-14.  Set  test  harness  Heading  SI  Ml 
TOR  to  090.0  . 

a.  Control  indicator  DISTANCE  TOGO 
counter  does  not  change. 

h.  Control  tndicatoi  auxiliary  DIS- 
TANCE CROSS  TRACK  counter  incre.i'er 

RIGHT. 

c.  Indicator  pointer  should  be  at  90  . 

d.  Indicator  cross  track  bar  slow!;, 
moving  lelt. 

e.  Indicator  DISTANCE  TO  GO  counter 
does  not  change. 

6-15.  Set  test  harness  HEADING  S1MV  LA- 
TOR  to  180.0’. 

h.  Control  indicator DISTANt  ETOOO 
counter  should  increase. 

b.  Control  indicator  auxiliary  DIS- 
TANCE CROSS  TRACK  counter  should  not 

change 

c.  Indicator  pointer  should  be  at  180  . 

d.  Indicator  cross  track  bar  does  not 
change  (not  necessarily  positioned  at  center 
of  scale). 


Figure  5.  Example  Page  from  TO  Fitild  Maintenance  Instructions  for 
Computer  Set,  Navigational 
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Section  VIII 


0)  f«H 

GO  HJ 
CTJ  C 


a-  o 


T.O.  5N1-3-8-2-TS-1 


3.  On  air  navigation  multiple  indicator,  observe  that  OFF  flag  is 
not  visible. 


4 . 

On 

navigational 

computer,  observe  that 

blower 

is 

ope rat i ng . 

5. 

On 

test  harness, 

, set  115V  AC  switch  to 

OFF. 

6 . 

On 

control  indicator,  observe  that  OFF 

l ndicator 

1 s 

lit. 

7. 

On 

test  harness. 

, set  115V  AC  switch  to 

ON  . 

8. 

On 

test  harness, 

, observe  that  26  Vac  power  is 

pr< 

St- 

nt  at  to s t 

points  B12  (hot)  and  E6(return). 

9.  On  test  harness,  adiust  HEADING  TRACK  SIMU1.AT0R  readout 
sliqhtly  if  necessary  for  precise  null  (minimum  ac  voltage)  between 
test  points  D6  and  D7 . Use  ME-6(  ) /U  Electronic  Voltmeter  for  null 
indication . 

3-5.  DISTANCE  TO  GO  SENSE  AND  READOUT 

1.  On  control  indicator,  observe  that  STAGE  I DISTANCE  TO  GO  read- 
out is  decreasing. 

2.  On  air  navigation  multiple  indicator,  observe  that  DISTANCE  TO 
GO  readout  is  decreasing. 

3.  On  air  navigation  multiple  indicator,  observe  that  DISTANCE  TO 
GO  readout  agrees  ( ±002)  with  STAGE  I DISTANCE  TO  GO  readout  on  con- 
trol indicator. 

4.  On  control  indicator,  sot  stage  selector  switch  to  STAGE  II 
ACTIVE.  Then,  observe  that  STAGE  II  DISTANCE  TO  GO  readout  is 
decreasing . 

5.  On  air  navigation  multiple  indicator,  observe  that  DISTANCE  TO 
GO  readout  is  decreasing. 

6.  On  air  navigation  multiple  indicator,  observe  that  DISTANCE  TO 
GO  readout  agrees  ( i 0 0 2 ) with  STAGE  II  DISTANCI  TO  GO  readout  on 
control  indicator. 

7.  On  control  indicator,  set  STAGE  I and  STAGE  II  DESIRED  TRACK 
ANGLE  readouts  to  180.0.  Then,  observe  that  STAGE  II  DISTANCE  TO  GO 
readout  is  increasing. 

8.  On  control  indicator  set  stage  selector  switch  to  STAGE  I 
ACTIVE.  Then,  observe  that  STAGE  I DISTANCE  TO  GO  readout  is 
increasing. 


3-2 


Figure  7. 


Example  Page  from  LTTA  Bench  Checkout  and  Troubleshooting 
Procedures  for  Computer  Set  AN/ASN-35  (Section  3 of  LTTA) 
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T.O.  5N 1 - 3-8-2-TS- 1 


Table  3-1.  Computer  Set  AN/ASN-35  Troubleshooting  Crossroference 

(Continued) 


STKP 

FAILED 


INDICATION 


PROCEDURE 


3-5/2 
Cont ' d 


If  all  ac  voltaqes  are 
present  and  vary,  air 
navigation  multiple  indi- 
cator is  faulty.  Other- 
wise, control  indicator 
is  faulty.  Perform  con- 
trol indicator  bench 
checkout  procedure. 


3-5/3 


3-5/4 


3-5/5 


3-5  6 


On  air  navigation  multiple  indi- 
cator and  control  indicator, 
DISTANCE  TO  GO  readouts  do  not 
agree. 

On  control  indicator,  STAGE  II 
DISTANCE  TO  GO  readout  not  de- 
creasing . 


Figure  3-3. 


Control  indicator  faulty. 
Perform  control  indicator 
bench  checkout  procedure. 


On  air  navigation  multiple  indi- 
cator, DISTANCE  TO  GO  readout  not 
decreasing. 


Control  indicator  faulty. 
Perform  control  indicator 
bench  checkout  procedure. 


On  control  indicator  and  air  navi- 
gation multiple  indicator, 

DISTANCE  TO  GO  readouts  do  not 
agree . 


Control  indicator  faulty. 
Perform  control  indicator 
bench  checkout  procedure. 


3-5  7 


On  control  indicator,  STAGE  II 
DISTANCE  TO  GO  readout  not  in- 
creasing. 


3-5/8 


On  control  indicator,  STAGE  I 
DISTANCE  TO  GO  readout  not  in- 
creasing . 


On  test  harness,  check 
fo’r  a 15-  to  35-V  positive 
pulse,  35-  to  85-ms  dura- 
tion occurring  9 to  1 1 
times  per  minute  at  test 
points  F.12  (hot)  and  F.5 
(rtn).  If  pulse  is  pres- 
ent, control  indicator 
is  faulty.  Otherwise, 
navigational  computer 
track  resolvi r drive  300 
is  faulty.  Perform  tra  -k 
resolver  drive  bench 
checkout  procedure. 

Control  indicator  faulty. 
Perform  control  indicator 
bench  checkout  procedure. 


3-15 


Figure  8.  Example  Page  from  LTTA  Bench  Checkout  and  Troubleshooting 
Procedures  for  Computer  Set  AN/ASN-35  (Section  3 of  LTTA) 
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Disconnect  audio  oscillator 
from  it st  harness. 


On  test  harness , monitor  ac 
voltage  between  test  points 
A1}  and  Ah,  using  ME-t< 
Electronic  Voltmeter . 


on  control  indicator,  set 
STAGE  I DISTANCE  TO  GO 
readout  first  to  000  N.M. 
then  to  5 00  N.M. 


A»r  navigation  multiple 
indicator  faulty. 


Control  indicator  faulty. 
Perform  control  indicator 
bench  checkout  procedure. 


Figure  3-3.  Computer  Set  Troubleshooting  for  Failed  Bench 
Checkout  Procedure  3-5/3 


Figure  9 


Example  Page  from  LTTA  Bench  Checkout  and  Troubleshooting 
Procedures  for  Computer  Set  AN/ASN-35  (Section  3 of  LTTA) 


Ion 

test 

harness 

are  ac  nulls 

present 

bet  ween 

test  points  A5 

• me 

Ah 

who 

n STA. 

,E  I DISTANCE  TO 

.0 

i cat 

out 

on  control  indicator  1 

1 s 

set 

ft) 

ono  N 

M.  and  500  N.M.  | 

The  LTTAs  also  contain  functional  block  diagrams  as  additional  aids 
to  the  technician  although  use  of  these  diagrams  is  generally  not 
required.  LTTAs  are  similar  to  conventional  manuals  in  that  they  are 
prepared  for  the  more  experienced  technicians.  Technicians  are  assumed 
to  be  able  to  properly  choose  and  use  test  equipment,  locate  most  parts 
and  test  points  with  general  type  illustrations  and  read  and  interpret 
functional  block  diagrams. 

Examples  of  the  LTTA  format  for  intermediate  level  maintenance  were 
presented  in  Figures  7,  8,  and  9.  These  figures  provide  an  example  of 
how  the  LTTAs  are  used.  On  step  3-5.3  of  Figure  7,  if  the  technician 
found  disagreement  in  the  two  readings,  he  would  refer  to  the  trouble- 
shooting cross-reference  table  (Figure  8).  This  table  would  refer  him 
to  Figure  3-3  of  the  LTTA  (Figure  9).  This  figure  contains  the  logic 
tree  which  would  be  followed  in  arriving  at  the  fault  identification. 

Flight  line  LTTAs  follow  a similar  format  as  the  intermediate  LTTAs 
with  two  basic  exceptions:  (a)  the  checkout  procedure  is  in  a separate 

"Job  Guide  Manual"  (a  4-  by  8-inch  size  format),  and  (b)  the  illustra- 
tions provided  with  the  fllghtllne  manual  are  line  drawings  keyed  to 
the  text  rather  than  photographs  as  used  with  the  maintenance  level 
LTTAs.  The  illustrations  for  the  organizational  level  LTTAs  are  line 
drawings  similar  to  those  used  in  the  FPTAs.  They  provide  more  detail 
than  the  illustrations  used  with  the  intermediate  level  LTTAs  but  less 
detail  than  the  FPTA  illustrations. 

Fully  procedural lzed  troubleshooting  aids  provide  complete  step-by- 
step  instructions  for  both  checkout  and  troubleshooting  of  the  equipment. 
These  aids  are  designed  for  use  by  both  experienced  and  inexperienced 
technicians.  A complete  logical  troubleshooting  procedure  is  provided 
similar  to  the  LTTAs.  Here,  however,  all  specific  test  equipment  and 
tools  are  called  out  in  each  section  as  they  are  required.  Each  prime 
hardware  item  and  the  more  complex  test  equipments  and  tools  mentioned 
in  the  body  of  the  text  are  accompanied  by  a callout  number,  in 
parentheses,  keyed  to  an  illustration  of  the  items  appearing  on  that 
same  page  or  a facing  page.  For  test  equipment  the  illustrations  appear 
in  a fold-out  page  at  the  back  of  each  section  (Figure  15).  As  soon  as 
the  technician  becomes  familiar  with  the  test  equipment  he  need  no 
longer  refer  to  the  fold-outs.  The  FPTAs  are  complemented  by  maintenance 
support  information  manuals.  These  manuals  contain  Instructions  on  test 
equipment  operation,  special  testing  procedure  and  wiring  diagrams. 

They  are  not  required  for  use  with  the  FPTAs  but  are  available  to  the 
technician  if  desired.  To  facilitate  use  of  the  aids  by  personnel  with 
experience  on  the  equipment,  the  basic  checks,  expected  readings,  and 
tolerances  are  underlined.  Underlined  information  permits  the 
experienced  technician  to  rapidly  scan  the  text  and  thus  focus  his  attention 
on  pertinent  information  only. 

Examples  of  the  FPTA  format  covering  the  same  malfunction  dealt 
with  in  the  LTTA  example  (non-agreement  of  DISTANCE-MILES  and  STAGE  I 
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DISTANCE  TO  GO  counters)  appear  in  Figures  10  through  15.  The  beginning 
of  the  checkout  procedure  is  presented  in  Figure  10.  The  FPTA  page 
(where  the  malfunction  is  encountered)  is  shown  as  Figure  11.  The 
proceduralized  troubleshooting  procedure  to  be  followed  begins  on  the 
page  shown  in  Figure  12.  (Troubleshoot  malfunction  No.  37  continue j on 
the  pages  shown  in  Figures  13  and  14.)  The  fold-out  referred  to  in 
Step  1 (Figure  10)  appears  at  the  end  of  the  checkout  procedure  and  is 
depicted  in  Figure  15. 


DEVELOPMENT  OF  FPTA  AND  LTTA  DOCUMENTATION 

The  FPTAs  for  both  organizational  and  intermediate  level  maintenance 
were  developed  under  contract  with  Kentron  Hawaii,  Ltd.  The  FPTAs  were 
developed  in  accordance  with  the  requirements  of  the  draft  specification 
contained  in  AFHRL-TR-73-43(I) . This  specification  provides  detailed 
requirements  for  the  content  and  format  of  the  data.  In  addition,  it 
requires  that  a thorough  task  identification  and  analysis  be  performed 
and  that  specific  "subproducts"  be  generated  during  the  task  analysis 
and  data  development.  The  subproducts  are  required  to  ensure  that  the 
contractor  follows  development  procedures  which  will  provide  high  quality 
data,  i.e.,  data  that  are  complete  and  provide  the  technician  with  all 
the  information  he  needs.  The  procedures  also  ensure  that  the  data  are 
written  at  a level  of  detail  suitable  for  the  intended  user. 

The  following  procedures  were  used  in  the  development  of  the  FPTAs: 

1.  Development  of  Task  Identification  Matrix  (TIM).  The  TIM  contains 
a listing  of  all  components  and  indicates  which  maintenance  tasks 
(e.g.,  remove,  replace,  troubleshoot)  are  performed  on  the 
component.  The  TIM  provides  the  FPTA  developer  with  a means  of 
determining  the  tasks  for  which  procedures  must  be  developed. 

2.  Development  of  User  Description.  The  user  description  contains 
information  on  the  aptitudes,  training  and  experience  of  the 
intended  user.  This  information  is  obtained  through  Air  Force 
records  and  interviews  with  potential  users  and  their  supervisors. 

3.  Development  of  JPA/Training  Trade  Off  Ground  Rules.  The  JPA/ 
Training  Trade  Off  Ground  Rules  are  developed  by  representatives 
of  the  data  developer  and  training  developer.  They  are  based 

on  the  user  description  and  training  policies  and  requirements 
established  by  the  procuring  agency.  They  provide  the  basis 
for  determining  which  tasks  proceduralized  instructions  will  be 
developed  for  and  the  level  of  detail  to  be  used. 

4.  Development  of  the  Level-of-Detail  Guide.  The  Level-of-Detall 
Guide  provides  the  individual  writing  the  procedure  with 
guidelines  for  determining  the  level  of  detail  to  be  provided 
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T.O.  5N1-3-TS-3 

CHECKOUT  AND  TROUBLESHOOT  CONTROL 

INDICATOR  C-3819A/ASN-35  (CONT) 

Inspect  Control  Indicator. 

CAUTION 

If  returning  to  this  page  from 

a troubleshooting  step,  be  sure: 

a.  All  disconnected  wires 
and  cables  are  recon- 
nected. 

b.  All  removed  components 
are  reinstalled. 

1.  Open  foldout  (p.  1-59). 


Do  not  overlook  importance  of 
visual  inspections.  If  you  do 
not  know  how  to  inspect  com- 
ponents, refer  to  T.O.  MSIM 
GENL. 

Replace  defective  components, 
as  discovered,  and  continue 
checkout.  Refer  to  IPB  (T.O. 
5N5-12-2-4)  for  part  number. 

2.  Perform  thorough  visual  inspection 
of  control  indicator  (12).  Look 
for: 

a.  Burned  or  scorched  components. 

b.  Cracked,  disconnected,  and 
broken  wires. 

c.  Bent,  broken,  and  missing  pins 
(13)  and  terminals  (14). 

3.  Inspect  control  indicator  (12)  for 
foreign  matter.  Remove  foreign 
matter,  as  required. 


Check  Counter  Assemblies. 


Be  sure  to  rotate  L/R  counter 
assembly  (6)  through  its  full 
count  999  LEFT  and  RIGHT  999. 

4.  Operate  controls  (7),  (8),  and 
(11)  to  rotate  counter  assemblies 
(6),  (9),  and  (5)  through  their 
full  count.  Check  that  counters 
(6),  (9),  and  (5)  rotate  freely. 

If  not,  inform  supervisor. 

Check  Torque  Transmitter  Continuity. 

5.  Disconnect  P5405  (15).  Using  VOM, 
check  that  resistance  between 
P5401-30  (3)  and  P5401-32  (4)  is 

8 to  10  ohms.  If  not,  go  to 
p.  1-12  (malfunction  No.  1). 

6.  Connect  P5405  (15).  Disconnect 
P5406  (16). 

7.  Usinq  VOM,  check  that  resistance 
between  P5401-30  (3)  and  P5401-32 
(4)  is  8 to  10  ohms.  If  not,  go 
to  p.  1-12  (malfunction  No.  2). 
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Figure  10.  Example  Page  from  FPTA  Checkout  and  Troubleshoot  Procedures  for 
Navigational  Computer  AN/ASN-35 
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T.O.  5N1-3-TS-3 

CHECKOUT  ANO  TROUBLESHOOT  CONTROL 
INDICATOR  C-3819A/ASN-35  (CONT) 

Check  Zero  Switches  SW5403,  SW5404. 

52.  Set  FUNCTION  switch  (65)  to  STAGE 
RECYCLING.  Rotate  ACTIVE/OFF  switch 
(36)  to  STAGE  I ACTIVE/STAGE  II  OFF. 

53.  Set  AUTO-MAN-OFF  switch  (40)  to 
AUTO.  ACTIVE/OFF  switch  (36) 
should  remain  STAGE  I ACTIVE/STAGE 
II  OFF.  If  not,  go  to  p . 1-35. 

54.  Rotate  ACTIVE/OFF  switch  (36)  to 
STAGE  I OFF/STAGE  II  ACTIVE. 
ACTIVE/OFF  switch  (36)  should  re- 
main STAGE  I OFF/ STAGE  II  ACTIVE. 

If  not,  go  to  p.  1-36. 

Check  DISTANCE  TO  GO  Counter  Operation. 

55.  Set  FUNCTION  switch  (65)  to  ALONG 
TRACK.  Set  DISTANCE-MILES  counter 
(66)  to  000. 

56.  Set  DISTANCE  TO  GO  counters  (38) 
and  (39)  to  000  000.  Rotate  ACTIVE/ 
OFF  switch  (36)  to  STAGE  I ACTIVE/ 
STAGE  II  OFF. 

57.  Set  COUNTER  SETTING  switch  (63) 
to  FAST  INCR.  Return  switch  (63) 

to  0,  as  required,  to  stop  DISTANCE- 
MILES  counter  (66)  at  750.  Check 
that: 

a.  STAGE  I DISTANCE  TO  GO  counter 
(39)  indicates  750.  If  not, 

go  to  p.  1-37. 

b.  ALONG  TRACK-MILES  counter  (31) 
indicates  750  tOOl.  If  not, 
go  to  p.  1-39. 

58.  Return  DISTANCE-MILES  counter  (66) 
to  000.  Return  STAGE  I DISTANCE 
TO  GO  counter  (39)  to  000. 

59.  Set  COUNTER  SETTING  switch  (63)  to 
FAST  DECR.  Return  switch  (63)  to 

0,  as  required,  to  stop  DISTANCE-  .. 
MILES  counter  (66)  at  250.  Check 
that: 

a.  STAGE  I DISTANCE  TO  GO  counter 
(39)  Indicates  250.  If  not, 
go  to  p.  1-41. 

1-10 


b.  ALONG  TRACK-MILES  counter  (31) 
indicates  250  +001.  If  not, 
go  to  p.  1-39. 

60.  Rotate  ACTIVE/OFF  switch  (36)  to 
STAGE  I OFF/STAGE  II  ACTIVE.  Re- 
turn DISTANCE-MILES  counter  (66) 
to  000. 

61.  Set  COUNTER  SETTING  switch  (63)  to 
FAST  INCR.  Return  switch  (63)  to  0, 
as  required,  to  stop  DISTANCE-MILES 
counter  (66)  at  750.  Check  that: 

a.  STAGE  II  DISTANCE  TO  GO  counter 
(38)  indicates  750.  If  not, 

go  to  p.  1-43. 

b.  ALONG  TRACK-MILES  counter  (31) 
indicates  750  +001.  If  not, 
go  to  p.  1-45. 

62.  Return  DISTANCE-MILES  counter  (66) 
to  000.  Return  STAGE  II  DISTANCE 
TO  GO  counter  (38)  tq  000. 

63.  Set  counter  setting  switch  (63)  to 
FAST  DECR.  Return  switch  (63)  to 
0,  as  required,  to  stop  DISTANCE- 
MILES  counter  (66)  at  250.  Check 
that: 

a.  STAGE  II  DISTANCE  TO  GO  counter 
(38)  indicates  250.  If  not, 

go  to  p.  1-47. 

b.  ALONG  TRACK-MILES  counter  (31) 
indicates  250  +001.  If  not, 
go  to  p.  1-45. 

Check  Stage  Switching. 

64.  Set  FUNCTION  switch  (65)  to  STAGE 
RECYCLING. 

65.  Set  DISTANCE  TO  GO  counters  (38) 
and  (39)  to  002  002.  Rotate  ACTIVE/ 
OFF  switch  (36)  to  STAGE  I ACTIVE/ 
STAGE  II  OFF.  Set  AUTO-MAN-OFF 
switch  (40)  to  AUTO. 

66.  Decrease  STAGE  I DISTANCE  TO  GO 
counter  (39)  to  000.  Check  that 
ACTIVE/OFF  switch  (36)  changes  to 
STAGE  I OFF/STAGE  II  ACTIVE.  If 
not,  go  to  p.  1-48. 

67.  Depress  and  hold  PRESS  TO  TEST 
switch  (44).  Using  RMS  voltmeter. 


Figure  11.  Example  Page  from  FPTA  Checkout  and  Troubleshoot  Procedures  for 
Navigational  Computer  AN/ASN-35 


35 


Usinq  VOM,  check  that  resistance 
between  SU54Q2-C7  (3)  arT3 
SW5402-CR  (2)  is  1 or  Jess. 
not,  go  to  step  27T.  if  VoM  i nd i • 
cates  1 ohm  or  less,  go  to  next 
step. 


Troubleshoot  Malfunction  No.  37 


SUMMARY  NOTE 


When  DISTANCE-MILES  counter 
indicates  250,  STAGE  I 01  ST 
ANCE  TO  GO  counter  does  not 
Indicate  250. 


Using  VOM,  check  that  C5411  (1 ) is 
good.  If  not,  go  to  step  2l . If 
C5411  (1)  is  good,  go  to  step  24. 


Set  ON/OFF  switch  (54)  to  OFF  and 
unplug  P5402  (11).  Using  VOM, 
check  that  115V-1/2A  SLO-BLO  fuse 
(53)  is  goocT  Replace  fuse  (53), 
as  required,  and  go  to  step  2. 

Using  VOM,  check  that  resistance 
between  P5402-2  (10)  and  P54.02- 3 
(9)  is  60  to  80  ohms.  If  not,  go 
to  step~7^  If  VOM  indicates  60  to 
80  ohms,  go  to  next  step. 

Remove  L/H  and  R/H  along  track  coun 
ter  assemblies  (16)  and  (15)  (vol. 
6,  p.  3-1)  and  go  to  next  step. 


19.  Replace  TX5401,  transmitter,  tor 
que  (17)  (vol.  6,  p.  3-1)  and  qo 
to  p.  1-1. 


20.  Replace  SW5402,  switch  (7)  (Vol. 
6,  p.  3-1 ) .and  go  to  p.  1-1 . 

21.  Replace  C5411,  capacitor,  0.02  uF 
(1 ) and  go  to  p.  1-1 . 

22.  Replace  C5404,  capacitor,  0.05  uF 
(8)  and  go  to  p.  1-1. 

23.  Replace  C5412,  capacitor,  0.02  uF 
(6)  and  go  to  p.  1-1 . 


24.  Using  control  indicator  wiring 
diagram  (vol.  6)  check  wiring. 
Repair  or  replace  defective  wires 
and  go  to  o.  1-1. 


TROUBLESHOOT  MALFUNCTION  NO.  36 
ENOS  HERE 


Figure  12.  Example  Page  from  FPTA  Checkout  and  Troubleshoot  Procedures  for 
Navigational  Computer  AN/ASN-35 


CHECKOUT  AND  TROUBLESHOOT  CONTROL 
INDICATOR  C-3819A/ASN-35  (CONT) 


9.  Using  VOM,  check  that  resistance 
between  J5406-J  (10)  and  J$40(TT 
(9)  is  60  to  80  ohms.  If  not,  gi 
to  step  14.  If  VOM  indicates  60 
to  80  ohms,  go  to  next  step. 


Troubleshoot  Malfunction  No.  37  (Cont) 

4.  Check  that  right  counter  magnet 
assembly  K5405  (4)  adjustment 
is  good.  If  not,  go  to  step  11. 

If  adjustment  is  good,  go  to  next 
sten. 


Using  control  indicator  wiring  dia- 
gram (vol.  6)  check  wiring.  Repair 
or  replace  defective  wires  and  go 
to  p.  1-1. 


11.  Adjust  riqht  counter  magnet  assem 
bly  K5405  (4)  (vol.  6,  p.  5-1) 
and  go  to  step  15. 


Check  that  riaht  counter  magnet 
core  K5405  (3)  adjustment  is  good 
If  not,  "go  to  step  12.  If  adjust 
ment  is  good,  go  to  next  step. 


Inform  supervisor  malfunction  is 
mechanical  and  located  in  L/H  along 
track  counter  assembly  (2). 


Using  VOM,  check  that  resistance 
between  SM5402-C10  (5)  and 
SW5402-C11  (6)  isH  ohm  or  less. 
If  not,  qo  to  step  13.  If  VOM 
indicates  L ohm  or  less,  go  to  m 
step. 


8.  Disconnect  P5406  (8)  and  go  to  next 
step. 


Figure  13.  Example  Page  from  FPTA  Checkout  and  Trouble 
Navigational  Computer  AN/ASN-35 


Troubleshoot  Malfunction  No.  38 


12.  Adjust  right  counter  magnet  core 
K5405  (3)  (vol . 6,  p.  5-1)  and 
go  to  step  15. 

13.  Replace  SW5402,  switch  (7)  (vol.  6 
p.  3-1 ) and  go  to  p.  1-1 . 

14.  Replace  K5405,  magnet  counter 
assembly  (4)  (vol.  5,  p.  3-1)  and 
go  to  step  15. 

15.  Instal  L/H  and  R/H  along  track 

counter  assemblies  (2)  and  (1) 
(vol.  6,  p.  3-1).  Plug  in  P5402 
(11).  Set  ON/OFF  switch  (54)  to 
ON  and  go  to  p.  1-1.  ' 


SUMMARY  NOTE 


When  DISTANCE-MILES  counted  indi- 
cates 750,  STAGE  II  DISTANCE  TO  GO 
counter  does  not  indicate- 750. 


1.  Set  ON/OFF  switch  (54)  to  OFF. 
Using  VOM,  check  that  115V-1/2A 
SL0-BL0  fuse  (53)  is  good.  Re7 
place  fuse  (53),  as  required,  am 
go  to  next  step. 

2.  Unplug  P5402  (11)  and  go  to  next 
step. 


Using  VOM,  check  that  resistance 
between  P5402-1  (12)  and  P54027? 
(13)  is  60  to  80  ohms.  If  not, 
to  step  8.  If  VOM  indicates  60 
80  ohms,  go  to  next  step. 


TROUBLESHOOT  MALFUNCTION  NO.  37 
ENDS  HERE 


Figure  14 


Example  Page  from  FPTA  Checkout  and  Troubleshoot  Procedures  for 
Navigational  Computer  AN/ASN-35 
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in  the  instructions.  For  example,  it  provides  the  basis  for 
determining  whether  an  instruction  should  say  "Using  VOM,  check, 
that  28  Vdc  is  present  at  TP05,"  or  should  say  "Using  Simpson 
260,  set  dc  scale  to  50,  check  for  28  Vdc  at  TP05."  The  level 
. of  detail  guide  is  based  on  the  user  description  and  JPA/training 

trade  off  ground  rules. 

5.  Development  of  Checkout  and  Troubleshooting  Procedures. 

Checkout  and  troubleshooting  procedures  are  developed  for  each 
hardware  item  listed  in  the  TIM  as  found  in  troubleshooting. 

This  is  accomplished  by  first  preparing  a list  of  components 
and  their  failure  modes  (all  of  the  ways  that  a component 

can  fail.)  A list  of  malfunction  symptoms  is  then  prepared 
which  tests  all  of  the  observable  symptoms.  This  listing 
identifies  all  components  whose  failure  could  cause  each 
symptom.  Once  all  symptoms  are  known,  a checkout  procedure  is 
developed  to  exercise  the  system  and  detect  any  symptom  which 
may  be  present.  Action  trees  are  developed  to  isolate  the  cause 
of  each  malfunction  symptom  identified  in  the  checkout.  The 
action  tree  specifies  a logical  sequence  of  checks  designed  to 
isolate  the  fault  in  the  fewest  number  of  steps. 

6.  Formatting  of  Checkout  and  Troubleshooting  Procedures. 

After  the  checkout  procedure  and  action  trees  have  been  developed, 
they  are  placed  in  the  format  required  by  the  specification.  This 
format  arranges  the  information  in  three  sections.  The  first 
section.  Preliminary  Information  Page,  provides  the  technician 
with  information  required  to  do  the  job,  including  tool  and  test 
equipment  requirements,  supplies,  and  personnel  requirements. 

The  preliminary  information  page  is  followed  by  the  checkout 
procedure  which  is  presented  in  a step-by-step  format.  The 
checkout  procedure  is  followed  by  the  troubleshooting  procedure 
which  is  also  in  the  step-by-step  format.  Both  the  checkout 
and  troubleshooting  are  supported  by  and  keyed  to  detailed 
illustrations  of  the  referenced  components. 

The  requirements  of  the  specification  were  strictly  followed  with 
the  following  exceptions: 

1.  Not  all  subproducts,  required  by  the  specification,  were  produced 
in  the  exact  format  specified.  Some  of  the  specified  formats 
proved  to  be  too  time  consuming  and  cumbersome.  To  conserve 
resources,  the  contractor  was  to  provide  the  information  for 
in-process  reviews  in  altered  formats.  Although  the  formats 
were  altered,  the  analysis  procedures  used  to  develop  the 
information  were  not  altered. 

2.  The  specification  requires  that  all  illustrations  be  placed  on 
the  facing  or  same  page  as  the  procedure  referencing  the 
illustration.  It  was  realized  early  in  the  development  process 


40 


that  this  requirement  would  result  in  an  excessive  number  of 
pages.  It  was  observed  that  many  of  the  illustrations  of 
test  equipment  would  be  repeated  from  page  to  page  without 
change.  Therefore,  to  reduce  page  requirements  and  unnecessary 
repetition,  the  contractor  was  permitted  to  use  fold-out  pages 
to  present  these  illustrations.  Use  of  the  fold-outs  reduced 
the  total  number  of  pages  by  approximately  50  percent. 

In-process  reviews  were  held  throughout  the  development  process  at 
points  specified  in  the  specification.  A strong  emphasis  was  placed  on 
these  reviews  to  ensure  high  quality  data. 

The  contractor  was  required  to  accomplish  a 100  percent  validation 
of  all  procedures.  Dedicated  equipment  was  provided  to  the  contractor 
at  his  plant  for  this  purpose. 

The  LTTAs  for  organizational  level  maintenance  were  developed  by 
Westinghouse  Corporation.  The  LTTAs  were  developed  as  part  of  the 
Air  Force  technical  order  improvement  program.  Under  this  program  Job 
Guide  Manuals  and  LTTAs  were  developed  for  organization  level  equipment 
maintenance  of  the  C-141.  These  aids  are  presently  in  operational  use. 
The  LTTAs  were  developed  according  to  MIL-M-38800A  as  modified  by  a 
requirement  for  a complete  fault  analysis.  Specific  developmental 
procedures  or  techniques  are  not  prescribed  by  this  specification,  but 
are  left  to  the  discretion  of  the  contractor.  Thus,  emphasis  is  placed 
by  the  procuring  agency  on  the  product  rather  than  on  both  the  process 
and  product  which  is  the  case  in  the  development  of  FPTAs.  Dedicated 
aircraft  were  available  for  hands-on  analysis  and  tryout  of  the  checkout 
and  troubleshooting  procedures. 

The  intermediate  level  LTTAs  were  developed  specifically  for  this 
project  by  Lockheed  Electronics  Company  under  contract  with  AFHRL.  The 
aids  were  developed  according  to  MIL-M-38800A  as  modified  by  a require- 
ment for  a task  analysis.  As  with  the  organizational  level  LTTAs, 
specific  developmental  procedures  were  not  specified  or  required.  The 
contractor  was  given  freedom  to  use  his  own  procedures.  Complete  AN/ 
APN-147  and  AN/ASN-35  systems  were  provided  to  the  contractor  at  his 
plant  for  use  in  the  task  analysis  and  validation  of  procedures.  A 
100  percent  validation  of  all  checkout  procedures  and  selected 
troubleshooting  trees  was  required. 

In-process  reviews  were  held  periodically  at  the  contractor's 
facility  to  ensure  the  accuracy  and  adequacy  of  the  data.  The  reviews 
were  held  at  approximately  the  same  intervals  as  in-process  reviews 
for  the  FPTA  development.  Approximately  the  same  level  of  effort 
was  expended  by  the  procuring  agency  in  supervising  the  development 
of  FPTAs  and  LTTAs. 


Section  III 
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ACCURACY  ANALYSIS 


I 


l 


One  of  the  goals  of  the  project  was  to  evaluate  the  relative  accuracy 
of  FPTAs  and  LTTAs  and  their  adequacy  for  isolating  equipment  faults.  This 
was  accomplished  for  two  purposes: 

!•  To  provide  data  relative  to  the  question  of  whether  the  stringent 
FPTA  development  procedures  produce  more  accurate  data. 

2.  To  provide  a pool  of  potential  problems  for  use  in  the  evaluation. 
This  pool  provided  a list  of  problems  for  which  it  was  known 
that  FPTAs,  LTTAs,  and  TOs  provide  adequate  procedures  and 
information  for  isolating  the  faults.  The  job  performance 
tests  were  built  to  measure  troubleshooting  performance  on  15 
problems  from  this  list. 


SELECTION  OF  PROBLEMS 

The  equipment  problems  against  which  the  documentation  types  were 
evaluated  were  selected  after  an  intensive  review  of  worldwide  Air  Force 
maintenance  actions  on  the  AN/APN-147  and  AN/ASN-35  systems  for  calendar 
year  1974.  The  data  contained  in  these  reports  were  summarized  to  provide 
identification  of  maintenance  areas  which  appeared  to  be  logical  and 
representative  candidates  for  development  of  troubleshooting  test  problems. 
In  an  attempt  to  provide  field  verification  of  the  correctness  of  the  data 
summarizations,  the  results  were  discussed  in  detail  with  maintenance 
personnel  at  Charleston  AFB,  South  Carolina.  (This  base  was  not  included 
in  the  field  evaluation  effort).  As  a result  of  the  Charleston  AFB 
interviews,  detailed  information,  not  available  from  the  AFLC  reports, 
was  obtained  on  the  specific  parts  or  components  which  were  contributing 
most  to  the  maintenance  problems  of  the  AN/APN-147  and  the  AN/ASN-35. 

This  information  was  considered  in  selection  of  the  specific  trouble- 
shooting test  problems.  A total  of  30  malfunctions  were  identified 
(15  AN/APN-147  and  15  AN/ASN-35).  Each  of  the  malfunctions  selected  was 
located  on  an  applicable  system  schematic  to  verify  that  the  problems 
selected  were  satisfactorily  distributed  throughout  the  system. 


METHOD  OF  CHECKING  FOR  ACCURACY 

Each  of  the  candidate  problems  was  inserted  in  working  equipment 
to  determine  the  operational  symptoms  associated  with  the  failed 
equipment  part  or  component.  Once  this  action  had  resulted  in  identi- 
fication of  the  known  effects  of  the  fault,  a systematic  evaluation  of 
the  accuracy  of  the  technical  documentation  types  in  supporting  the 
isolation  of  the  faults  was  undertaken.  This  was  undertaken  using 
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each  type  of  data  to  isolate  the  fault.  Records  were  maintained  on  the 
errors  found  in  the  data  and  whether  the  troubleshooting  procedure  led 
to  isolation  of  the  problem.  This  approach  made  use  of  the  same  (primary 
and  test)  equipment  and  aids  as  used  in  the  field  evaluation  portion  of 
the  experiment. 

It  was  necessary  to  use  substitutes  for  some  problems  to  assure 
compatibility  with  the  degree  of  simulation  fidelity  employed  in  the 
Air  Force  simulations  of  a C-141A  cockpit  and  intermediate  (shop)  level 
maintenance  benches.  As  an  example,  the  technical  documentation  guiding 
isolation  of  certain  of  the  faults  originally  selected  directed  the 
troubleshooter  to  the  Horizontal  Situation  Indicator  (HSI).  The  HSI  was 
represented  by  a nonfunctioning  simulation  of  the  Air  Force  trailer  and 
thus  the  particular  fault  could  not  be  resolved  by  the  technical  documen- 
tation/equipment simulation  combination.  In  an  instance  such  as  this, 
a new  candidate  fault  was  selected  and  the  ability  of  the  types  of 
technical  documentation  to  find  it  was  verified.  Any  editorial  or 
content  problems  with  the  candidate  types  of  technical  documentation 
discovered  as  a result  of  following  this  process  were  reported  to  the 
Air  Force  technical  monitor  for  correction.  Examples  qf  some  of  the 
specific  problems  encountered  with  each  of  the  data  types  are  specified 
in  the  following  sections. 


FINDINGS 

The  results  of  the  accuracy  analysis  are  presented  in  the  following 
sections. 


FPTAs 

A number  of  typographical  and  technical  errors  were  found  in  the 
FPTAs.  These  may  be  classified  into  four  categories. 

1.  Inadequate  illustration.  In  some  cases  the  illustrations  were 
unclear  or  the  call-out  arrows  were  not  properly  placed  to 
clearly  identify  the  referenced  component. 

2.  Incorrect  steps  or  values.  In  some  instances  the  technician  is 
directed  to  go  to  an  incorrect  step.  In  other  Instances 
incorrect  tolerances  and  meter  readings  are  given.  A sufficient 
number  of  such  errors  were  encountered  to  indicate  a need  for 
increased  quality  control  in  these  areas. 


3. 


Inadequate  tolerances.  In  some  instances  specific  values  are 
specified  for  some  readings  or  checks.  This  does  not  allow 
sufficient  tolerances  for  variations  between  individual  systems. 
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identified  as  bad.  For  example,  steps  5 and  6 on' page  1-4 
of  T.O. 5N1-3-TS-1  specifies  that  the  "distance  to  go"  indicator 
must  count  down  to  "000"  in  10  minutes.  Individual  differences 
in  equipment  result  in  this  countdown  seldom  occurring  at 
exactly  10  minutes.  A tolerance  of  9.5  to  10.5  minutes  would 
have  alleviated  this  problem. 

4.  Clerical  errors.  Clerical  errors  were  found  to  a degree 

throughout  the  FPTAs.  Errors  of  this  type  can  be  relatively 
minor  or  can  cause  the  procedure  to  be  completely  ineffective. 
Increased  quality  control  in  this  area  is  needed. 


LTTAs 

The  errors  found  in  the  LTTAs  were  similar  in  nature  to  those  found 
in  the  FPTAs. 

1.  Incorrect  tolerances.  In  some  instances  referenced  values  for 
readings  gave  no  permissible  tolerances,  causing  technicians  to 
incorrectly  classify  components  as  being  faulty. 

2.  Unclear  intent.  It  was  unclear  to  the  technicians  that  a 
detailed  check  procedure  followed  the  minimum  checkout  of 
equipment.  Instances  of  unclear  designation  of  specific 
switches  or  counters  were  found.  A further  instance  of  unclear 
intent  was  found  where  technicians  were  given  multiple  symptoms. 
These  were  not  adequately  identified  to  permit  cross-reference 
with  the  symptoms  contained  in  T.O. 141A-2-8J6-2  (Checkout). 
Finally,  word  descriptions  of  the  same  symptom  appearing  at 
different  places  in  the  LTTA  were  inconsistent,  again  leading 

to  confusion  in  intent. 

3.  Clerical  errors.  Clerical  errors  were  found  to  a degree 
throughout  the  LTTA.  The  errors  can  have  impacts  ranging 
from  negligible  to  completely  disruptive  on  success  with  the 
procedure.  Increased  quality  control  is  needed. 

Figure  16  contains  a summarization  of  the  documentation  problems  by 
number  and  nature  found  for  each  of  the  15  troubleshooting  problems 
used  in  the  experimental  evaluation  portion  of  this  study. 

In  perusing  Figure  16,  the  reader  is  cautioned  against  assigning 
an  equal  severity  to  each  indicated  problem.  In  such  a comparison, 

FPTAs  would  have  been  credited  with  29  problems  and  LTTAs  with  only  14. 
The  conclusion  might  be  erroneously  made  that  LTTAs  were  twice  as 
problem-free  as  were  FPTAs.  FPTAs  are  written  to  a much  greater  level 
of  detail  than  are  LTTAs  and  the  problems  noted  were  frequently  of  a 
more  minor  nature  than  those  with  LTTAs.  Each  document  type  had 
problems  and  these  problems  had  varied  levels  of  severity.  The  number 


Number  of 
Problems  with 
Document 


9 

FPTA 

LTTA 

Nature  of 
Problem 


Incorrect  instruction  (2) 
Switch  identification  (1) 


Misleading  instruction  (2) 

No  tolerance  for  reading  (1) 

Incorrect  reference  (1) 

No  tolerance  for  reading  (1) 
Unclear  procedure  (1) 

Unclear  meaning  (1) 


Incorrect  reference  (1) 
Unclear  reference  (1) 

Switch  identification  (1) 
Unclear  wording  (1) 


Misleading  instruction  (2) 

No  tolerance  for  reading  (1) 


Insufficient  tolerances  (1) 
Incorrect  identification  (1) 

Insufficient  tolerances  (1) 
Unclear  procedure  (4) 


Inaccurate  reference  (1) 


Incorrect  reference  (1) 


Switch  identification  (2) 


Insufficient  tolerances  (1) 
Incorrect  instruction  (1) 
Incomplete  instruction  (1) 


Figure  16.  Summarization  of  Problems  Encountered  with  FPTA  and  LTTA 
Documentation  for  Troubleshooting  Problems  Used  in  Study 


Troubleshooting 
Problem  Number 

Document 

Type 

Number  of 
Problems  with 
Document 

Nature  of 
Problem 

10 

FPTA 

3 

Incorrect  instruction  (3) 

LTTA 

0 

11 

FPTA 

1 

Incorrect  reference  (1) 

LTTA 

2 

Unclear  instructions  (2) 

12 

FPTA 

6 

Insufficient  tolerances  (4) 
Unclear  instruction  (2) 

LTTA 

0 

13 

FPTA 

0 

LTTA 

0 

14 

FPTA 

0 

LTTA 

0 

15 

FPTA 

1 

Unclear  instruction  (1) 

LTTA 

1 

Unclear  procedure  (1) 

Figure  16.  Summarization  of  Problems  Encountered  with  FPTA  and  LTTA 
Documentation  for  Troubleshooting  Problems  Used  in  Study 
(Concluded) 
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of  errors  found  is  not  greater  than  would  normally  be  expected  in  any  TO. 
All  errors  found  can  be  corrected  relatively  easily.  If  sufficient  funds 
had  been  available  for  a more  complete  verification,  it  is  likely  that 
most  of  the  errors  would  have  been  detected  and  corrected.  In  all 
instances,  however,  it  was  concluded  that  these  problems  were  soluble. 

Problems  from  the  original  list  of  30  candidate  faults  which  were 
not  selected  for  inclusion  were  excluded  for  a variety  of  reasons.  The 
major  reasons  for  rejection  were  equipment  centered  (required  equipment 
not  included  in  the  mockup  or  equipment  that  was  not  available).  Four  of 
the  15  nonselected  problems  were  rejected  on  the  basis  of  inaccuracies  or 
other  problems  with  the  FPTA  or  LTTA. 

In  these  four  instances,  results  were  as  follows: 

For  two  problems  the  LTTA  permitted  the  location  of  the  fault,  but 
the  FPTA  directed  the  technician  along  troubleshooting  paths 
which  resulted  in  identification  of  a gpod  part  as  being  bad.  In 
the  remaining  two  instances  neither  the  FPTA  nor  LTTA  detected  the 
existence  of  the  fault,  (i.e.,  the  checkout  procedure  led  to  the 
conclusion  that  the  system  was  operating  satisfactorily). 
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Section  IV 


COST  COMPARISON 


A major  consideration  in  determining  if  either  FPTAs  or  LTTAs  should 
be  adopted  for  use  with  existing  and  future  Air  Force  systems  concerns 
their  cost.  More  specifically,  how  much  more  would  have  to  be  invested 
to  provide  either  FPTAs  or  LTTAs  as  an  adjunct  to  existing  TOs  or  as  a 
complete  replacement. 

Although  it  was  a project  goal  to  provide  complete  answers  to  these 
questions,  the  task  proved  to  be  more  difficult  than  anticipated.  First 
of  all,  this  study  was  concerned  only  with  technical  data  for  trouble- 
shooting. It  was  impossible  to  equate  FPTAs  and  LTTAs  with  the 
conventional  TOs  because  of  differences  in  content,  differences  in  data 
base  development,  and  time  of  development.  First,  troubleshooting 
data  in  the  TO  was  mixed  with  other  types  of  data  throughout  the  manual 
(it  was  for  this  reason,  among  others,  that  the  more  compact  FPTAs  and 
LTTAs  were  developed).  Secondly,  the  existing  TO  did  serve  as  a partial 
data  base  for  the  FPTA  and  LTTA.  A true  comparison  would  be  possible 
only  if  all  three  types  of  data  were  developed  independently  on  a new 
system.  Finally,  the  TO  was  developed  a number  of  years  ago  when  labor 
rates,  and  dollar  costs*  were  different.  The  TO  had  also  been  updated 
and  modified  a number  of  times  since  initial  development.  For  these 
reasons  subsequent  analyses  were  restricted  to  a comparison  of  FPTAs 
and  LTTAs  only.  The  results  are  reported  below  along  with  some  general 
comments  on  additional  considerations  to  be  made  in  comparing  technical 
data  systems. 


COMPARISON  OF  FPTA  AND  LTTA  COSTS 

Since  the  FPTAs  and  LTTAs  cover  the  same  tasks  and  were  developed 
from  the  same  data  base,  a direct  comparison  of  costs  is  possible.  The 
costs  are  presented  in  Table  1.  They  are  presented  in  terms  of  cost 
per  page  unit  and  total  cost.  A page  unit  is  defined  as  the  area 
contained  in  a standard  8 1/2-  by  11-inch  page.  The  LTTAs  for  organiza- 
tional level  maintenance  are  printed  in  the  4-  by  8- inch  format.  The 
number  of  pages  has  been  converted  to  page  units  to  permit  a meaningful 
comparison. 

As  may  be  seen  from  Table  1,  a total  of  880  page  units  were  required 
to  present  the  FPTAs  and  623  page  units  were  required  to  present  the 
LTTAs.  The  largest  number  of  page  units  are  required  for  the  intermediate 
level  aids.  A total  of  809  page  units  were  required  for  the  FPTAs  while 
only  545  page  units  were  required  for  the  LTTAs.  Thus,  264  or  48  percent 
more  pages  were  required  to  present  the  FPTA  at  the  intermediate  level. 
Examination  of  the  data  suggests  that  the  additional  pages  are  required 
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primarily  as  a result  of  the  extensive  use  of  illustrations  in  the  FPTA. 
The  more  detailed  instructions  in  the  FPTA  also  contribute  to  the 
requirement  for  more  pages. 


TABLE  1.  COMPARISON  OF  COSTS  EXPERIENCED  FOR  THE  DEVELOPMENT 
OF  FPTAs  AND  LTTAs  FOR  THE  AN/APN-147  AND  AN/ASN-35 


Technical  Data 
Format 

Cost  Per 
Page  Unit 

Number  of 
Page  Units 

Total 

Cost 

FPTA 

Organizational 

203.56 

71 

14,451 

Intermediate 

203.56 

809 

164,680 

880 

179,131 

LTTA 

Organizational 

97.34 

78 

7,593 

Intermediate 

203. 80 

545 

111,076 

623 

118,669 

At  the  organizational  level,  the  order  of  magnitude  is  reversed 
with  slightly  more  page  units  required  for  the  LTTA  than  for  the  FPTA. 
Seventy-eight  ages  were  required  for  the  LTTA  and  71  pages  were  required 
for  the  FPTA.  It  would  be  expected  that  the  FPTA  would  require  more  page 
units.  Examination  of  the  data  reveals  the  reason  for  more  pages  being 
required  for  the  LTTA.  The  reason  is  that  the  LTTAs  for  the  organizational 
level  contain  schematics.  The  FPTAs  do  not  have  schematics  (schematics 
for  use  with  the  FPTA  are  included  with  the  job  guide  manuals  discussed 
in  a later  section).  When  the  LTTA  schematics  are  excluded  from  the 
page  count,  it  is  observed  that  59  page  units  are  required  to  present 
the  LTTAs.  Thus,  the  FPTA  requires  12  more  page  units  to  present  the 
troubleshooting  procedures. 

From  Table  1 it  may  also  be  seen  that  the  cost  per  page  unit  was 
$203.56  for  FPTAs,  $97.34  for  organizational  level  LTTAs  and  $203.80. 
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for  intermediate  level  LTTAs2.  The  total  cost  for  FPTAs  was  $179,131. 

The  total  cost  for  LTTAs  was  $118,669.  Thus,  the  FPTAs  cost  significant- 
ly more  (53%)  than  the  LTTAs.  There  are  two  basic  factors  which 
contribute  to  higher  cost  for  FPTAs.  They  are  higher  developmental 
costs  and  higher  production  costs.  The  higher  developmental  costs  result 
from  the  more  thorough  and  more  rigidly  specified  task  analysis  required 
for  the  development  of  the  FPTA.  The  FPTA  task  analysis  requires  some 
procedures  (such  as  analysis  of  the  capabilities  of  the  user  and 
development  of  the  training/ JPA  trade  off  ground  rules)  that  are  not 
required  in  the  LTTA  task  analysis.  In  addition,  the  FPTA  task  analysis 
requires  that  specific  "subproducts"  (such  as  the  test  equipment  and 
tool  use  form)  be  developed  and  delivered  in  a specified  format. 

The  second  factor  which  contributes  to  the  higher  cost  of  FPTAs  is 
higher  production  costs.  The  higher  production  costs  result  primarily 
from  the  larger  number  of  pages  required  for  the  FPTA.  As  indicated 
above,  the  larger  number  of  pages  is  due  primarily  to  the  extensive  use 
of  illustrations  (at  least  one-third  of  each  page  is  used  for  illustra- 
tions). Thus,  considering  the  cost  to  develop  the  illustrations  and  the 
cost  of  producing  the  additional  pages,  the  use  of  illustrations  account 
for  a significant  portion  of  the  higher  cost  of  FPTAs. 


ADDITIONAL  COST  CONSIDERATIONS 

The  above  discussion  relates  to  the  cost  of  data  for  troubleshoot- 
ing only.  Neither  FPTAs  nor  LTTAs  provide  a complete  system  of 
technical  data  for  maintaining  the  system.  The  TO,  on  the  other  hand, 
provides  a complete  system  of  technical  data  for  maintaining  all  aspects 
of  the  system.  (This  difference  in  coverage  is  another  reason  that  it 
was  not  considered  appropriate  to  directly  compare  TO  costs  with  FPTA 
and  LTTA  costs.) 

A complete  system  of  technical  data  based  on  the  FPTA  approach 
would  require  two  additional  types  of  data.  Documentation  presenting 


2 In  comparing  the  costs  for  developing  FPTAs  and  LTTAs,  two 
factors  must  be  considered.  The  LTTAs  for  organizational  level  were 
developed  under  a large  contract  to  prepare  LTTAs  and  JGMs  for  the  entire 
C-141  aircraft.  Thus,  overhead  costs,  etc.,  are  spread  out  over  a larger 
cost  base.  Secondly,  the  FPTAs  and  LTTAs  were  developed  by  contractors 
in  different  areas  of  the  country  where  labor  rates  differ  significantly. 
The  FPTAs  were  developed  by  Kentron  Hawaii,  Ltd.,  in  Dallas,  Texas,  a 
relatively  low  labor  cost  area.  The  intermediate  level  LTTAs  were 
developed  by  Lockheed  Electronics  Company  in  Plainfield,  New  Jersey. 

If  comparable  labor  rates  had  been  charged  for  developing  both  sets  of 
data,  it  is  likely  that  the  cost  per  page  unit  would  have  been 
significantly  lower  for  producing  the  LTTAs. 
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theory  of  operation,  schematics  and  other  miscellaneous  information  would 
be  required  for  use  in  troubleshooting  those  unusual  problems  not  isolated 
by  the  FPTA.  Also,  job  guide  manuals  (JGMs)  would  be  required  for  non- 
troubleshooting tasks  such  as  remove,  replace,  align,  and  adjust.  JGMs 
and  schematics  were  developed  for  use  with  the  FPTAs  on  the  AN/APN-147 
and  AN/ASN-35.  These  aids  cost  a total  of  $32,400  (270  pages  at  $120 
per  page).  Thus,  the  total  FPTA-JGM  package  cost  a total  of  $211,531. 
Theory  of  operation  type  information  was  not  developed  for  use  with  the 
systems.  There  is  no  data  available  for  estimating  the  cost  of  this 
data,  or  the  cost  of  a complete  FPTA  based  technical  data  system  for 
the  AN/APN-147  and  AN/ASN-35. 

A complete  system  of  technical  data  based  on  LTTAs  would  also 
require  two  additional  types  of  technical  data.  Documentation  presenting 
theory  of  operation  and  other  miscellaneous  information  would  be 
required  for  use  in  troubleshooting  those  unusual  problems  not  isolated 
by  the  LTTA.  (Schematics  were  included  in  the  LTTAs.)  Also,  some  type 
of  documentation  would  be  required  for  nontroubleshooting  tasks  such 
as  remove,  replace,  align,  and  adjust.  This  type  of  data  would  be 
presented  in  JGMs  similar  to  those  developed  for  use  with  the  FPTAs. 

The  JGMs  developed  for  use  with  the  FPTAs  were  not  developed  under  the 
specification  (MIL-M-38800A)  that  would  be  used  in  the  development  of 
the  JGMs  for  use  with  LTTAs.  Thus,  the  JGMs  would  differ  somewhat  in 
format  and  may  not  be  based  on  the  same  type  of  thorough  task  analysis 
required  for  development  of  the  JGMs  produced  for  use  with  the  FPTAs. 

Since  JGMs  were  not  made  for  use  with  the  LTTAs,  there  is  no  basis  for 
a direct  cost  comparison  with  the  JGMs  developed  for  use  with  the  FPTAs. 
The  cost  for  these  JGMs  should  be  no  higher  than  the  cost  for  the  FPTA 
style  JGMs  and  may  be  somewhat  less  expensive.  Whether  the  JGMs 
developed  according  to  MIL-M-38800A  would  be  less  expensive  would  depend 
on  whether  a task  analysis  is  required  and  the  thoroughness  of  the  task 
analysis.  Since  theory  of  operation  data  was  not  developed  for  use  with 
the  LTTAs,  there  is  no  basis  for  estimating  the  cost  to  develop  this 
data.  However,  there  is  no  reason  to  believe  that  the  cost  for 
developing  theory  of  operation  information  for  use  with  LTTAs  would 
differ  markedly  from  the  cost  to  develop  the  data  for  use  with  FPTAs 
or  conventional  TOs. 
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EXPERIMENTAL  PROCEDURES 


This  section  describes  the  procedures  used  and  the  results  of  the 
experimental  evaluation  portion  of  the  project.  The  basic  goal  of  the 
experimental  evaluation  was  to  collect  hard  data  relative  to  the 
effectiveness  of  each  type  of  data  for  supporting  troubleshooting  by 
personnel  of  varying  levels  of  experience.  The  experiment  was 
designed  to  provide  answers  to  questions  listed  on  page  15  (Section  1). 

It  was  desired  to  evaluate  the  effectiveness  of  the  three  types  of 
data  for  supporting  maintenance  personnel  with  three  levels  of 
experience  - apprentice  technicians  with  no  experience,  technicians  with 
less  than  six  months  experience  on  the  systems,  and  technicians  with 
more  than  six  months  experience  on  the  system.  Technical  orders  are  not 
designed  for  independent  use  by  3-level  technicians  and  3-level 
technicians  are  not  expected  to  be  able  to  troubleshoot  on  their  own 
using  TOs.  On  the  basis  of  these  factors  it  was  determined  that  it  would 
not  be  appropriate  or  necessary  to  test  the  apprentice  technicians 
using  TOs.  As  a result  of  this  decision,  it  was  necessary  to  use  two 
different  but  similar  experimental  designs.  In  the  first  design, 
apprentice  technicians  were  tested  using  two  types  of  data,  FPTAs  and 
LTTAs.  In  the  second  design,  experienced  technicians  were  tested  using 
all  three  types  of  data.  Data  collection  for  the  apprentice  subjects 
was  accomplished  at  Keesler  Technical  Training  Center  (KTTC) . Data 
collection  for  the  experienced  technicians  was  accomplished  at  three 
Military  Airlift  Command  bases  - McChord,  Travis,  and  McGuire.  The 
procedures,  test  facilities,  materials  used,  and  results  are  described 
in  the  following  sections. 


TEST  FACILITIES  AND  MATERIALS 

Training  Materials.  To  ensure  that  observed  differences  in  perform- 
ance were  due  to  the  technical  data  and  not  due  to  the  technician’s 
not  knowing  how  to  use  the  required  test  equipment,  training  programs 
were  developed  for  18  pieces  of  test  equipment  required  to  maintain  the 
systems  (see  Figure  17).  The  programs  were  prepared  as  individual 
self-taught  packages.  The  instructions  are  presented  in  a programmed 
text  format.  The  time  required  to  complete  each  program  ranges  from 
approximately  15  minutes  to  4 hours. 

Training  programs  were  also  developed  to  train  the  technicians 
to  use  FPTAs  and  LTTAs.  These  programs  were  prepared  in  a self-taught, 
programmed  text  format  similar  to  that  used  for  the  equipment  training 
programs.  The  programs  included  practice  exercises  in  which  the  techni- 
cians "performed"  several  checkout  and  troubleshooting  tasks  using  the 
test  benches  in  an  inactive  (power  off)  state.  The  exercises  had  the 
dual  purpose  of  developing  familiarity  with  the  data  and  with  the  test 
benches . 
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1 410B  VTVM  Vacuum  Tube  Voltmeter 


2 260  VOM  Volt  Ohmmeter 

3 545B  Oscilloscope 

4 5245L  Electronic  Counter  and  Plug-ins 

5 TV2  Tube  Tester 

6 TS-148  Radar  Test  Set 

7 803  Differential  VTVM 

8 200CD  Audio  Oscillator 

9 URM-25D  Signal  Generator 

10  TS-1100/U  Test  Set,  Transistor 

11  CMA-544  Doppler  Simulator 

12  CMA-546  Doppler  Generator 

13  3322-900  Computer  Cards  Tester 

14  3322-902  Computer  Drives  Tester 

15  3322-905  Navigational  Tester 

16  3322-901  Computer  Relay  Chassis  Tester 

17  SG299B/U  Signal. Generator  

18  Kay  Sweep  Generator 

Figure  17.  List  of  18  Items  of  Test  Equipment  Available 
for  Use  with  Prime  Equipment 


Job  Performance  Tests.  Job  performance  tests  were  developed  to 
measure  the  performance  of  technicians  on  15  troubleshooting  problems. 

The  15  problems  were  selected  to  be  representative  (in  terms  of  difficulty) 
of  problems  routinely  occurring  in  the  test  systems.  The  problems  were 
selected  from  the  problems  used  to  evaluate  the  accuracy  of  the  data. 

They  were  selected  from  those  problems  for  which  the  FPTA  and  LTTA 
procedures  were  found  to  be  accurate.  The  tests  are  administered  by 
inserting  a malfunction  in  the  system  and  observing  the  performance  of 
the  technician  troubleshooting  the  problem.  Performance  is  evaluated 
in  terms  of  whether  the  problem  is  solved,  time  required,  and  spare  parts 
used.  The  job  performance  tests  are  provided  in  Volume  III  of  this 
report. 

Attitude  Data.  Questionnaires  were  developed  to  measure  the 
attitudes  of  the  technicians  toward  each  type  of  data.  Basically  two 
types  of  questionnaires  were  developed — a mid-experiment  questionnaire 
and  a post-experiment  questionnaire.  The  mid-experiment  questionnaire 
was  given  immediately  following  the  use  of  one  type  of  data.  It  asked 
the  technician  to  rate  the  data  on  factors  such  as  ease  of  understanding 
and  suitability  for  different  levels  of  maintenance.  The  post-experiment 
questionnaire  asked  the  technician  to  compare  the  data  on  the  same 
factors.  A rating  scale  was  used  to  obtain  a measure  of  technician 
attitudes.  A representative  scale  is  shown  below: 


Very  difficult  No  particular  Very  easy  to 

to  understand  difficulty  in  understand 

and  follow  following  and  follow 

The  technician  is  asked  to  indicate  by  marking  the  approximate  location 
on  the  scale  which  represents  his  feelings.  The  scale  is  scored  by 
measuring  the  distance  of  the  response  from  the  left  side.  Thus,  the 
higher  the  score  the  more  positive  the  response. 

Ancillary  Data  Collection.  In  addition  to  the  attitude  data  collected 
additional  questionnaires  and  inventories  were  administered  to  collect 
demographic,  aptitude,  attitude,  and  opinion  information.  These  data  were 
not  used  in  the  present  analysis  but  were  collected  for  use  in  future 
studies  and  analysis. 

Test  Facilities.  The  performance  test  approach  requires  dedicated 
access  to  the  prime  equipment  for  extensive  periods  of  time.  It  was 
realized  that  it  would  not  be  possible  to  obtain  sufficient  access  to 
test  benches  at  operational  bases  without  causing  an  unacceptable  degree 
of  interference  with  their  mission.  To  ensure  adequate  access  to  the 
equipment  at  all  test  locations,  a 40-foot  Air  Force  van  was  equipped 
with  bench  mockups  of  the  AN/APN-147  and  AN/ASN-35  to  provide  a trans- 
portable test  facility.  It  was  also  recognized  that  it  would  be 
extremely  difficult  to  obtain  adequate  access  to  C-141  aircraft  for  use 
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with  the  organizational  level  tests.  Thus,  a mockup  of  the  C-141 
cockpit  and  the  avionics  equipment  bays  of  the  C-141  was  developed 
and  installed  in  the  test  van.  The  cockpit  mockup  provided  a means 
of  simulating  the  flightline  troubleshooting  tasks  without  requiring 
access  to  an  aircraft.  It  consists  of  mockups  of  all  instrument  panels 
in  the  cockpit.  All  AN/ APN-147  and  AN/ASN-35  components  are  "live"  and 
function  in  the  same  manner  as  in  the  actual  cockpit.  All  other 
components  on  the  instrument  panels  are  represented  by  photographs. 

The  trailer  was  transported  to  each  test  site.  At  Keesler  an 
additional  computer  bench  and  a radar  bench,  installed  in  KTTC  class- 
rooms and  used  in  technical  training,  were  made  available  for  use  in 
running  subjects. 

Stopwatches  were  provided  for  the  experimenters,  and  for  the  subject 
when  required.  In  addition,  a standard  tool  kit  including  fuses,  bulbs, 
and  jumper  wires  was  provided.  Certain  problems  required  the  use  of 
shorting  connectors;  these  were  made  available.  Additionally,  a 
resolver  bridge  balancing  Jig  (3173-918)  and  labeling  tags  were  provided 
as  special  materials  for  problem  11  and  an  inclinometer  was  supplied 
for  problem  12.  Further  detail  on  the  materials  required  for  each 
problem  may  be  found  in  Volume  III  (Test  Administrator's  Guide)  of  this 
report. 


PROCEDURES  AT  KEESLER  TTC 

A counterbalanced,  repeated  measures  experimental  design  was  used 
for  the  data  collection.  In  this  design  each  subject  performed  seven 
problems  using  FPTAs  and  seven  problems  using  LTTAs.  One-half  of  the 
subjects  (technicians)  performed  their  seven  problems  with  FPTA  first. 

The  other  half  of  the  subjects  completed  their  seven  LTTA  problems  first. 
The  job  performance  tests  were  used  to  measure  the  subjects’  performance 
in  terms  of  successful  completion  of  the  problem,  spare  parts  consumed, 
and  time  to  perform.  


Subjects.  Eighteen  airmen  graduates  of  KTTC  Course  No.  3ABR32834 
served  as  subjects  for  this  portion  of  the  study.  The  21  October  and 
4 November  1975  graduating  classes  were  used.  The  airmen  were  retained 
at  Keesler  AFB  for  a 2-week  period  beyond  course  graduation  to  serve  as 
subjects  in  the  experiment.  During  the  period  of  involvement  in  the 
study,  the  airmen  were  exempted  from  additional  duties  and  were  accorded 
the  status  of  graduates,  a designation  which  led  to  additional  privileges 
over  those  typically  enjoyed  by  students. 

The  subjects  were  either  Airman  or  Airman  First  Class.  Two  of  the 
subjects  were  female  and  16  were  male.  All  subjects  were  classified  as 
having  high  aptitude  for  electronics  iulntenance  (AQE  scores  of  80  or 
higher).  The  training  received  by  the  subjects  in  Course  No.  3ABR32834 
Included  18  weeks  of  training  in  basic  electronic  fundamentals  and  18 
weeks  of  theory  oriented  training  on  representative  sets.  Training  on  the 


AN/APN-147  and  AN/ASN-35  was  received  as  part  of  the  sets  training.  This 
training  was  3 weeks  in  duration.  It  was  primarily  theory  oriented.  Some 
opportunity  for  hands-on  practice  of  troubleshooting  tasks  was  provided. 
However,  this  practice  was  very  limited. 

Procedures.  Random  assignment  procedures  were  used  to  determine 
the  order  of  presentation  of  problems  and  the  type  of  data  used  with  each. 
This  was  accomplished  by  first  randomly  assigning  each  subject  to  one 
of  two  groups:  one  group  used  FPTA  first;  the  other.  LTTA  first.  Each 
subject  was  then  randomly  assigned  to  1 of  18  counterbalanced  sequences 
of  problem  presentation  order.  Finally,  in  the  subject-problem  pairing 
process,  the  sequence  of  exposure  to  level  of  maintenance  was  systema- 
tically counterbalanced.  The  counterbalancing  and  problem  assignment 
procedures  ensured  that  each  problem  was  performed  an  equal  number  of 
times  with  each  type  of  data  and  to  control  for  possible  order  effects 
in  the  experiment. 

With  the  KTTC  subjects,  14  of  the  15  equipment  problems  were  used. 
Problem  9 was  excluded  (see  Figure  3,  AFHRL-TR-76-74 (II) , for  problem 
identification).  Exclusion  of  this  problem  permitted  a balanced 
presentation  of  problems  with  the  KTTC  subjects;  seven  problems  with 
each  type  of  technical  documentation.  The  14  problems  used  were  composed 
of  four  organizational  maintenance  level  and  10  intermediate  maintenance 
level  problems.  This  division  was  selected  to  represent  the  approximate 
proportion  of  organizational  and  intermediate  level  maintenance  tasks 
found  in  the  operational  environment.  There  were  an  equal  number  of  radar 
and  computer  problems  at  the  organizational  and  intermediate  levels. 

The  problem  assignment  procedure  described  above  was  followed  in  an 
attempt  to  control  for  possible  order  effects  in  the  experiment. 

First  access  to  each  graduating  class  was  for  a one-half  hour  period 
on  the  day  of  graduation.  During  this  period,  members  of  the  evaluation 
team  were  introduced;  the  purpose  of  the  evaluation  was  described  to  the 
group;  a brief  tour  of  the  trailer  and  classroom  evaluation  areas  was 
conducted;  and  each  class  member  was  given  his  initial  assignment  for 
the  following  day. 

Days  one  and  two  of  the  2-week  period  of  availability  were  devoted 
to  test  equipment  proficiency  training.  The  morning  of  day  three  was 
devoted  to  either  FPTA  or  LTTA  training,  depending  upon  the  particular 
group  to  which  the  airman  was  assigned.  The  afternoon  of  day  three  was 
devoted  either  to  troubleshooting  of  problems  or  to  collection  of  ancillary 
factor  data,  again  depending  upon  the  group  to  which  the  airman  was 
assigned.  The  remainder  of  the  2-week  period  of  availability  followed 
predetermined  subject  schedules  (Table  1 and  Figures  7,  8,  and  9 of 
Volume  II). 
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At  the  point  of  cross-training  on  the  second  technical  data  format, 
each  subject  completed  an  opinion  questionnaire  on  the  documentation  format 
he  had  just  used.  The  subject  was  then  entered  into  cross-training  on  the 
use  of  the  second  type  of  technical  data,  and  exposed  to  troubleshooting 
problems  with  the  alternate  data  type  following  the  schedules  referred 
to  above. 

At  the  completion  of  the  second  set  of  experimental  problems,  the 
subject  was  given  two  opinion  questionnaires;  the  first,  concerning  the 
data  format  he  had  just  finished  using;  the  second,  an  opinion  questionnaire 
asking  him  to  compare  both  technical  documentation  formats. 

For  each  unique  troubleshooting  problem,  the  test  administrator  pre- 
pared the  equipment  by  completing  the  "pretest  setup"  section  of  the 
problem  contained  in  the  Test  Administrator's  Guide  (AFHRL-TR-76-74(IIl)) . 

In  addition,  the  presence  of  the  support  material  requirements  listed  in 
Section  IV  of  the  problem  was  verified.  At  this  point,  the  subject  was 
brought  into  the  troubleshooting  situation.  Following  the  instructions 
section  (Section  V)  of  the  guide,  the  subject  was  given  an  information 
and  instructions  sheet  and  asked  to  read  it.  Next,  he  was  given  a 
specific  instructions  sheet  and  a work  order  describing  the  equipment 
malfunction  he  was  to  troubleshoot. 

The  troubleshooting  evaluation  was  conducted  on  a one-to-one  basis 
(one  test  administrator  to  one  subject).  The  test  administrator 
recorded  data  on  student  performance,  obtaining  measures  of  time,  errors, 
parts  consumed,  successful  completion  of  each  section  (of  the  trouble- 
shooting procedure),  branching  followed,  correctness  of  the  branching, 
and  degree  of  success  in  locating  the  problem  (see  AFHRL-TR-76-74(IlI) 

Test  Administrator's  Guide,  for  sample  data  recording  forms). 

Since  the  adequacy  of  the  document  to  lead  the  technician  to  the 
equipment  component  causing  the  malfunction  was  the  basic  issue  under 
evaluation,  the  test  administrator  was  instructed  to  answer  no  procedural 
questions  while  the  test  problem  was  ongoing. 

PROCEDURES  AT  MAC  BASES 


A counterbalanced,  repeated  measures  experimental  design  similar 
to  that  used  for  the  Keesler  subjects  was  used  for  the  experimental 
subjects.  There  were  two  major  differences  in  the  designs.  The  MAC 
design  required  the  subjects  to  use  all  three  types  of  data.  Secondly, 
the  experienced  design  utilized  subjects  with  two  levels  of  experience: 
those  with  six  months  or  less  experience  on  the  equipment  and  those  with 
more  than  six  months  experience.  Each  subject  completed  five  problems 
with  each  type  of  data.  Data  use  was  counterbalanced  so  that  one-third 
of  the  subjects  used  FPTAs  first,  one-third  used  LTTAs  first  and  one- 
third  used  TOs  first.  Performance  was  measured  using  th£~«ame  Job 
performance  tests  and  procedures  as  used  at  Keesler  TTC. 
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Subject 8.  The  two  remaining  subject  groups  in  the  design  were 
obtained  from  Air  Force  enlisted  personnel  having  the  328X4  AFSC, 

Avionics  Inertial  and  Radar  Navigational  Systems  Specialist,  assigned 
to  operational  units  of  the  Military  Airlift  Command  (MAC)  and  performing 
maintenance  on  the  AN/ASN-35  and  AN/APN-147  systems  in  C-141A  aircraft. 

The  two  groups  were  classified  on  the  basis  of  length  of  experience 
with  the  C-141A  equipment.  Each  group  has  a size  of  n*18.  One  group 
consisted  of  airmen  with  six  months  or  less  experience  on  the  systems; 
the  second  group,  airmen  with  more  than  six  months  experience  on  the 
systems.  All  subjects  in  the  two  groups  were  male.  A 2-step  process  was 
used  to  select  the  technicians  to  serve  as  subjects  at  each  base.  First, 
a list  of  personnel  in  each  experience  category  was  obtained  from  the 
base.  Technicians  and  alternates  were  then  tentatively  selected  by  a 
random  process.  When  the  evaluation  team  arrived  at  the  base,  final 
arrangements  were  made  for  these  technicians  to  participate  in  the  study. 

If  a selected  airman  was  not  available  (due  to  leave,  TDY,  etc.)  an 
alternate  was  substituted.  Each  airman  selected  as  a subject  was  made 
available  for  a 2-week  period.  During  this  period,  the  airmen  continued 
to  work  their  normal  shifts  and  served  as  subjects  for  a 4-hour  period 
out  of  each  8-hour  shift. 

Procedures.  Within  each  experience  group  the  airmen  were  randomly 
assigned  to  one  of  three  subgroups  (six  airmen  each).  One  of  the  three 
types  of  technical  documents  to  br  initially  used  was  randomly  assigned 
to  each  group.  With  the  field-experienced  group,  all  15  equipment  problems 
were  used.  Thus,  each  technician  was  exposed  to  five  problems  with  each 
type  of  technician  documentation. 

The  order  of  exposure  of  level  of  maintenance  was  systematically 
counterbalanced.  In  addition,  within  each  experience  group,  18  series 
of  counterbalanced  sequences  of  problems  were  generated.  Each  subject 
was  randomly  assigned  to  one  of  the  18  sequences.  Each  problem  was 
performed  the  same  number  of  times  with  each  type  of  data. 

Airmen  selected  as  subjects  were  initially  provided  with  an 
introductory  letter  briefly  explaining  the  nature  of  the  study  in  which 
they  would  be  participating.  Airmen  were  then  given  the  pre-experiment 
questionnaires.  Next,  airmen  were  provided  with  a listing  of  the  test 
equipment  that  would  be  used  in  the  study  and  asked  to  estimate  their 
proficiency  with  each  item.  At  this  time,  each  airman  was  asked  to 
indicate  any  test  equipment  on  which  he  would  like  to  receive  refresher 
training.  Experience  with  the  test  equipment  and  observation  of  KTTC 
subject  interaction  with  the  equipment  led  to  a decision  to  routinely 
administer  proficiency  teats  on  four  items  of  test  equipment,  the  260 
volt  ohmmeter  (VOM),  the  410B  vacuum  tube  voltmeter  (VTVM),  the  545B 
oscilloscope,  and  the  TV2  tube  tester.  The  tests  were  administered 
regardless  of  the  level  of  proficiency  expressed  by  the  airmen  for 
these  items  of  test  equipment.  Remedial  training  was  given  to  those 
who  failed  to  demonstrate  proficiency  on  these  items  of  test  equipment. 
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Following  any  required  test  equipment  training,  the  airmen  were 
given  familiarization  training  with  the  new  documentation  type  if  their 
initial  exposure  was  FPTA  or  LTTA.  They  then  completed  the  performance 
tests  using  that  type  of  data.  Airmen  initially  assigned  to  TOs  went 
immediately  to  the  job  performance  tests  using  that  data. 

Upon  completion  of  the  assigned  problems,  airmen  were  given  a mid- 
experiment opinion  questionnaire  on  the  documentation  just  used  and  then 
were  transitioned  to  the  second  set  of  five  problems  with  the  next 
scheduled  documentation  type.  Upon  completion  of  problem  solving  with 
the  second  documentation  type,  the  airmen  were  again  given  a mid-experiment 
opinion  questionnaire  on  that  documentation  type  and  moved  to  the  final 
problem  set  with  the  remaining  documentation  type.  At  completion  of 
this  last  set  of  five  problems,  two  opinion  questionnaires  were  given. 

The  first  covered  the  document  type  just  used;  the  second  required  a 
comparison  of  the  three  types  of  documentation. 

Again,  the  test  administrator  followed  the  instructions  contained 
in  the  Test  Administrator's  Guide  (AFHRL-TR-76-74(III))  in  preparing 
for  each  experimental  problem.  Observation,  timing,  and  scoring  procedures 
were  identical  to  those  described  for  the  KTTC  group. 


ANALYSIS 

Main  analysis  for  both  the  KTTC  and  the  Military  Airlift  Command 
subjects  was  by  means  of  an  analysis  of  variance  of  the  data  collected  in 
a counterbalanced  order  of  presentation,  repeated  measures  design.  Follow- 
ing the  analysis  of  variance,  significance  tests  on  those  comparisons  of 
interest  to  the  study  were  calculated. 

Data  collected  on  the  KTTC  subjects  followed  a special  case 
repeated  measures  design.  In  each  case  with  this  design,  the  main 
effect  was  tested  with  the  factor  by  subjects'  interaction. 

Data  collected  on  the  Military  Airlift  Command  subjects  followed 
a Winer  (1962)  Case  I,  three-factor  repeated  measures  design.  Again, 
following  this  analysis,  significance  tests  on  the  comparisons  of 
interest  to  the  study  were  calculated. 
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Section  VI 


RESULTS  OF  EXPERIMENT 


INTRODUCTION 

The  study  results  are  presented  in  this  section  in  two  separate 
and  distinct  forms.  This  presentation  approach  was  selected  in  recogni- 
tion of  the  complex  nature  of  the  data  collected.  The  goal  of  this,  and 
of  any,  report  of  research  is  to  present  the  results  to  the  reader  in 
the  clearest  possible  manner,  and  yet  provide  sufficient  detailed  support 
to  allow  the  reader  to  independently  assess  the  material  presented. 

Ideally,  the  research  report  should  permit  the  reader  to  establish 
a level  of  confidence  that  he  is  willing  to  assign  to  the  results  and 
to  the  conclusions  drawn  from  the  obtained  results. 

Thus,  this  section  is  divided  into  a proportional  analysis  presenta- 
tion which  is  intended  to  portray  the  study  results  in  a clear  unencumbered 
form,  and  into  a statistical  analysis  presentation  which  treats  the  same 
results,  but  in  a different  form.  In  this  latter  section,  reporting  of 
a result  is  accompanied  with  results  of  the  underlying  statistical 
analysis. 

The  reader  interested  in  obtaining  a quick  overview  of  the  study 
results  can  obtain  this  information  by  reading  only  the  proportional 
analysis  presentation. 


PROPORTIONAL  ANALYSIS 

Keesler  Technical  Training  Center  Subjects 

Inspection  of  Figure  18  indicates  that  more  problems  were  solved 
with  the  FPTA  at  both  organizational  and  intermediate  levels  of  mainte- 
nance than  with  LTTA.  Within  document  types,  there  was  no  difference  in 
performance  at  organizational  and  intermediate  levels  when  using  the 
FPTA.  Use  of  the  LTTA  resulted  in  a difference  in  performance  between 
organizational  level  and  intermediate  level  troubleshooting  tasks  with 
better  performance  occurring  at  the  organizational  level. 

The  number  of  spare  parts  incorrectly  used  at  both  levels  of 
maintenance  for  each  type  of  documentation  is  shown  in  Figure  19. 

Initial  viewing  would  suggest  that  use  of  the  FPTA  resulted  in  the  use 
of  fewer  spare  parts.  The  statistical  analysis  revealed  that  this 
difference  in  mean  number  of  parts  used  was  not  large  enough  to  permit 
the  conclusion  that  a real  difference  exists.  Unlike  this  comparison, 
the  comparison  of  the  number  of  parts  incorrectly  used  at  the  two  levels 
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Figure  18.  Mean  Proportion  of  Problems  Solved  by  Keesler  Subjects 
by  Documentation  Type  and  Level  of  Maintenance 
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Figure  19.  Mean  Number  of  Parts  Used  by  Keesler  Subjects  by 
Documentation  Type  and  Level  of  Maintenance 


of  maintenance  shows  a large  enough  discrepancy  to  be  accepted  as  a real 
difference.  Fewer  parts  are  used  at  the  organizational  level  of 
maintenance.  Review  of  the  nature  of  the  troubleshooting  tasks  at  each 
level  of  maintenance  (organizational  and  intermediate)  would  lead  one  to 
expect  this  result. 

The  mean  time  required  to  troubleshoot  and  repaid  is  shown  in 
Figure  20.  It  can  be  seen  from  the  means  displayed  that  at  the  organi- 
zational level  of  maintenance  use  of  the  LTTA  as  an  aid  resulted  in  a 
shorter  mean  time  than  did  the  FPTA.  Insight  into  the  reason  for  this 
may  be  found  from  review  of  the  verbatim  comments  of  the  subjects  (see 
Table  A-6  Appendix  A).  The  attitude  expressed  was  that  the  FPTA 
forced  one  into  a more  lengthy  procedure.  However,  for  intermediate 
maintenance  the  order  was  reversed.  Here  use  of  the  FPTA  produced  a 
shorter  mean  time  to  repair. 

Military  Airlift  Command  Subjects 

The  mean  proportion  of  problems  solved  by  organizational  level  of 
maintenance  and  by  experience  level  of  technicians  is. depicted  in 
Figure  2L.  Inspection  of  the  bar  charts  contained  in  this  figure  shows 
that  at  the  organizational  level  of  maintenance  there  was  effectively  no 
difference  in  performance  for  any  of  the  documentation  types  regardless 
of  the  experience  levels  involved.  However,  this  same  result  did  not 
occur  for  intermediate  level  maintenance.  At  this  level,  performance 
differed  for  documentation  type.  For  the  TO,  performance  also  differed 
for  experience  level.  Technicians  with  more  than  six  months  system 
experience  performed  better  with  the  TO  as  an  aid  than  did  technicians 
with  less  than  six  months  experience  using  the  TO. 

At  the  intermediate  level  of  maintenance,  performance  was  better 
with  the  FPTA  than  with  the  LTTA,  regardless  of  the  experience  level 
of  the  technician.  However,  this  difference  in  performance  is  not 
large  enough  to  be  considered  a real  difference. 

Another  comparison  which  may  be  made  from  this  figure  is  that  of 
the  TO  against  the  proceduralized  troubleshooting  data  presentation 
(FPTA  and  LTTA)  for  intermediate  level  maintenance.  The  use  of  either 
the  FPTA  or  LTTA  resulted  in  better  troubleshooting  performance  regardless 
of  experience  level  of  the  technician.  This  difference  was  sufficiently 
large  to  be  considered  a real  (not  a chance)  difference. 


3 Total  time  required  to  troubleshoot  and  repair  a specific  problem 
was  obtained  for  each  technician  by  using  the  technician's  elapsed  time 
on  the  problem  and  adding  an  average  time  for  removal  and  replacement 
of  each  of  the  parts  identified  as  faulty.  Mean  times  in  Figure  20  are 
the  arithmetic  means  across  all  technicians. 
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Mean  Time  to  Troubleshoot  and  Repair  by  Level  of  Maintenance  and 
Document  Type  for  Keesler  Subjects 
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Figure  21.  Mean  Proportion  of  Problems  Solved  by  MAC  Subjects  by 

Experience  Level  at  the  Organizational  Level  of  Maintenance 
(continued  on  page  65) 
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Figure  21.  Mean  Proportion  of  Problems  Solved  by  MAC  Subjects  by 

Experience  Level  at  the  Intermediate  Level  of  Maintenance.  (Concluded) 


The  mean  number  of  parts  incorrectly  used  by  MAC  subjects  in  solving 
the  troubleshooting  problems  to  which  they  were  exposed  is  presented 
in  Figure  22.  It  is  apparent  from  this  bar  graph  that  a larger  number 
of  parts  were  used  for  intermediate  level  maintenance  tasks  than  for 
organizational  level  tasks.  This  result  would  be  expected  due  to  the 
nature  of  the  maintenance  involved  at  each  level.  At  the  organiza- 
tional level,  the  task  is  to  locate  the  line  replaceable  unit  which  has 
failed.  At  the  intermediate  level  the  task  can  be  to  locate  either  the 
faulty  subassembly  or  even  the  faulty  component.  Thus,  one  would  expect 
a greater  opportunity  for  identification  errors  at  the  intermediate  level. 
This  expectation  was  confirmed  by  the  data. 

Examination  of  Figure  22  also  reveals  that  there  is  very  little 
difference  in  the  number  oE  spare  parts  incorrectly  replaced  using 
FPTAs  and  LTTAs  at  the  intermediate  level.  However,  use  of  the  TO 
resulted  in  the  incorrect  use  of  more  spare  parts  than  the  use  of 
either  FPTA  or  LTTA. 

At  the  organizational  level,  performance  with  each  of  the  types  of 
technical  documentation  was  fairly  stable.  However,  again  use  of  TOs 
resulted  in  the  highest  number  of  parts  being  incorrectly  used.  The 
difference  in  mean  parts  incorrectly  used  for  organizational  level 
troubleshooting  was  not  large  enough  to  be  considered  a real  difference. 

The  mean  time  required  to  troubleshoot  and  repair  using  the  FPTA, 

LTTA,  or  TO  as  troubleshooting  aids  at  each  level  of  maintenance  is 
depicted  in  Figure  23.  Two  interesting  patterns  are  displayed  in  this 
figure.  First,  it  can  be  seen  that  the  pattern  of  times  required  at 
the  organizational  level  was  for  FPTA  to  take  the  longest  time,  followed 
by  LTTA,  with  TO  requiring  the  least  for  the  troubleshoot  and  repair 
task.  This  same  pattern  holds  regardless  of  the  experience  level  involved. 

At  the  intermediate  level,  the  pattern  is  exactly  reversed,  with 
FPTA-guided  troubleshoot  and  repair  taking  the  shortest  mean  time  and 
TO  requiring  the  longest  mean  time.  Again,  LTTA  time  fell  in-between 
FPTA  and  TO.  This  pattern  also  holds  true  regardless  of  the  experi- 
ence level  of  the  subjects. 

A significantly  greater  time  to  troubleshoot  and  repair  is  required 
when  working  at  the  intermediate  level.  This  result  is  again  expected 
for  the  same  reason  given  earlier  for  its  appearance  in  the  data  depicted 
in  Figure  22.  At  the  intermediate  level  of  maintenance,  the  technician 
is  working  at  a finer  level  of  detail  and  is  exposed  to  more  opportunity 
to  incorrectly  replace  parts  than  he  has  to  Incorrectly  replace  line 
replaceable  units  at  the  more  gross  organizational  level  of  maintenance. 
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STATISTICAL  ANALYSIS 


Kessler  Technical  Training  Center  Subjects 

The  effects  of  type  of  technical  documentation  on  troubleshooting 
performance  were  evaluated  by  means  of  an  analysis  of  variance  of  a 
special  case  repeated  measures  design.  An  administrative  decision  that 
KTTC  graduates  would  not  be  evaluated  on  TOs  resulted  in  the  data  set 
for  these  graduates  being  different  from  that  collected  with  the  field- 
experienced  technicians  of  the  Military  Airlift  Command.  For  this 
reason,  an  independent  analysis  of  the  Keesler  data  was  required. 

Three  separate  analyses  of  variance  were  computed  for  the  Keelser 
data,  one  where  number  of  problems  correctly  solved  was  the  criterion 
measure,  the  second  where  number  of  spare  parts  consumed  served  as  the 
criterion,  and  the  third  where  the  elapsed  time  to  isolate  and  repair 
the  fault  was  the  criterion.  The  0.05  level  of  confidence  was  selected 
as  the  level  of  probability  required  for  significance.  The  results  of 
the  analysis  of  variance  for  number  of  problems  correctly  solved  with 
each  type  of  technical  documentation  are  presented  in  Table  2;  the 
analysis  of  variance  results  for  number  of  spare  parts  in  Table  3;  and 
the  analysis  of  variance  results  for  time  to  repair  in  Table  4. 

An  attempt  was  made  to  expose  the  subjects  to  either  organizational 
or  maintenance  level  problems  in  a proportion  comparable  to  that  which 
they  could  be  expected  to  encounter  in  the  field.  Thus,  each  subject 
was  exposed  to  two  organizational  level  and  five  intermediate  level 
problems  with  each  of  the  two  documentation  types  (FPTA  and  LTTA)  used. 

For  the  first  analysis  of  variance  (number  of  problems  solved)  each  data 
point  has  been  transformed  to  compensate  for  the  unequal  weighting  of 
number  of  problems  at  each  level  of  maintenance.  The  transformation 
entails  converting  the  performance  score  to  the  proportion:  number 

solved/number  presented. 

For  computational  ease,  the  proportion  has  been  further  coded  by 
omission  of  the  decimal  and  division  by  10.  Thus,  the  possible  score 
any  subject  could  aicain  with  a documentation  type  at  a specific  level 
of  maintenance  ranges  from  0 to  10. 

Using  this  system  of  scoring,  a subject  correctly  solving  all 
problems  to  which  he  was  exposed  would  attain  a score  of  40.  With  the 
Keesler  subjects,  a range  of  scores  from  20  t«.  38  was  obtained.  The 
mean  scores  for  each  documentation  type  by  level  of  maintenance  were 
presented  in  Figure  18. 

Earlier  inspection  of  Figure  18  indicated  that  the  Keesler  subjects 
appeared  to  perform  better  with  the  FPTA  than  with  the  LTTA  documentation. 
Further,  unlike  the  LTTA,  performance  with  the  FPTA  was  maintained 
across  maintenance  levels. 
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TABLE  2.  SUMMARY  TABLE  FOR  ANALYSIS  OF  VARIANCE  OF  KEESLER 

TECHNICAL  TRAINING  CENTER  SUBJECTS,  NUMBER  OF  PROBLEMS 
CORRECTLY  SOLVED  WITH  FPTA  AND  WITH  LTTA,  OCTOBER  1975 


Source 

df 

SS 

MS 

F 

P 

Subjects 

17 

118.61 

Maintenance  Level 
Maintenance  Level  x 
Subjects 

1 

17 

14.22 

55.28 

14.22 

3.25 

4.38 

0.05<P<0. 10 

Document  Type 
Document  Type  x 
Subjects 

1 

17 

72.00 

75.50 

72.00 

4.44 

16.22 

<0.001 

Maintenance  Level  x 
Document  Type 

1 

14.22 

14.22 

3.82 

0.05<P<0. 10 

Maintenance  Level  x 
Document  Type  x 
Subjects 

17 

63.28 

3.72 

TOTAL 

71 

413.11 

71 


TABLE  3.  SUMMARY  TABLE  FOR  ANALYSIS  OF  VARIANCE  OF  KEESLER 
* TECHNICAL  TRAINING  CENTER  SUBJECTS,  NUMBER  OF  SPARE 


PARTS 

CONSUMED  WITH 

FPTA  AND  WITH 

LTTA, 

OCTOBER  1975 

Source 

df 

SS 

MS 

F 

P 

Subjects 

17 

53.57 

Maintenance  Level 
Maintenance  Level  * 
Subjects 

1 

17 

17.01 

44.24 

17.01 

2.60 

6.54 

<0.05 

Document  Type 
Document  Type  * 
Subjects 

1 

17 

5.01 

82.24 

5.01 

4.84 

1.04 

0.25<P<0.50 

Maintenance  Level  * 
Document  Type 

1 

1.13 

1.13 

<1 

Maintenance  Level  * 
Document  Type  * 
Subjects 

17 

57.12 

3.36 

I 


TOTAL 


71  260.32 


TABLE  4.  KEESLER  GRADUATES  - MEAN  TIME  REQUIRED  TO  TROUBLESHOOT 
WITH  TIME  ADJUSTED  TO  INCLUDE  REMOVE  AND  REPLACE  TIME 
FOR  PARTS  DESIGNATED  BY  SUBJECT  FOR  REPLACEMENT 


Source 

df 

SS 

MS 

F 

P 

Subjects 

17 

1899.07 

111.71 

Maintenance  Level 
Maintenance  Level  x 
Subjects 

1 

17 

386.42 

1687.26 

386.42 

99.25 

3.89 

0.05<P<0. 10 

Document  Type 
Document  Type  x 
Subjects 

1 

17 

317.52 

2397.59 

317.52 

193.98 

1.64 

0. 10<P<0. 25 

Maintenance  Level  x 
Document  Type 

1 

933.12 

933.12 

2.40 

0.05<P<0. 10 

Maintenance  Level  x 
Document  Type  x 
Subjects 

17 

6612.8 

388.99 

71 


TOTAL 


15,133.78 


That  the  difference  observed  by  inspection  of  Figure  18  for 
documentation  type  is  a real  difference  is  reflected  in  Table  2.  The 
F ratio  for  document  type  achieves  a value  which  is  accepted  as  reflecting 
a significant  difference.  Both  maintenance  level  and  the  interaction  of 
maintenance  level  with  documentation  type  show  a trend  toward  a signifi- 
cant difference  but  do  not  achieve  critical  value  of  F required  for 
rejecting  the  hypothesis  of  no  difference.  Inspection  of  Figure  18 
indicates  that  the  trend  toward  an  interaction  effect  is  apparently 
produced  by  the  failure  of  performance  to  remain  stable  at  the  inter- 
mediate level  of  maintenance  using  LTTA  documentation.  This  conclusion 
is  verified  by  a means  of  a least  significant  difference  (LSD)  analysis 
(Steel  and  Torrie,  1960).  With  the  LSD,  the  critical  value  required  for 
a significant  difference  at  the  0.05  level  of  significance  is  1.36.  Three 
degrees  of  freedom  for  comparison  exist.  Thus,  three  comparisons  are 
possible  with  the  data.  There  is  no  difference  for  the  mean  of  perform- 
ance with  FPTA  at  organizational  level  versus  FPTA  at  intermediate  level 
comparison;  therefore,  this  comparison  can  be  omitted.  The  comparison 
of  the  means  for  performance  with  FPTA  at  the  intermediate  level  versus 
LTTA  at  the  organizational  level  showed  no  real  difference.  The 
difference  of  means  for  performance  with  the  LTTA  at  the  organizational 
versus  the  intermediate  level  was  1.7.  Thus,  this  comparison  reveals  a 
real  difference  in  performance  with  influences  but,  in  itself,  is  not 
sufficiently  strong  to  result  in  a significant  overall  interaction  effect 
between  maintenance  level  and  documentation  type. 

The  data  presented  in  Table  3 depict  the  results  of  an  analysis  of 
variance  made  for  the  criterion  measure:  number  of  spare  parts  consumed 

with  each  of  the  documentation  types.  Inspection  of  these  data  reveals 
that  a significant  difference  exists  between  performance  with  each  of 
the  two  levels  of  maintenance.  The  mean  number  of  spare  parts  used  with 
each  document  type  at  each  level  of  maintenance  was  presented  in 
Figure  19.  From  inspection  of  this  bar  chart,  it  appeared  that  the 
effect  of  documentation  type  was  consistent  across  levels  of  maintenance; 
further,  that  the  difference  between  levels  of  maintenance  within  each 
documentation  type  was  greater  than  the  difference  between  documentation 
types.  These  contentions  are  supported  by  the  values  obtained  for  the 
F ratios  measuring  the  mean  effect  of  documentation  type,  the  effect  of 
maintenance  level,  and  the  Interaction  effect  between  maintenance  level 
and  documentation  type  which  appear  in  Table  3. 

Summary  data  on  the  analysis  of  variance  for  mean  time  required  to 
troubleshoot  and  repair  are  presented  in  Table  4.  No  effect  in  this 
analysis  achieved  statistical  significance.  However,  the  effect  of 
level  of  maintenance  showed  a strong  trend  toward  statistical  significance. 
The  factors  contributing  to  this  trend  were  graphically  presented  in 
Figure  20. 

As  part  of  the  study  plan,  each  subject  was  asked  to  provide  his 
opinion  for  selected  questions  about  each  documentation  type  after  he 
had  finished  using  that  document  and  to  make  a comparison  of  the  two 
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documentation  types  upon  completion  of  his  series  of  problems.  Opinion 
data  were  solicited  on  a Likert-type  scale  (refer  to  AFHRL-TR-76-74 (II) , 
Appendices  A,  B,  and  E-l).  For  case  of  comparison,  the  scale  associated 
with  each  question  was  uniform  in  length  (125  mm)  and  each  subject 
response  was  recorded  as  a millimeter  value  representing  the  distance  of 
his  response  from  the  left  (negative  valence)  side  of  the  scale.  Summary 
information  on  the  results  of  the  opinion  questionnaires  is  contained  in 
Appendix  A to  this  volume.  An  illustration  of  the  information  content 
is  found  in  the  example  of  the  opinion  expressed  by  Keesler  subjects 
toward  the  use  of  either  the  FPTA  or  LTTA  for  troubleshooting  equipment. 
The  mean  score  values  for  the  opinions  expressed  by  the  Keesler  subject 
group  are  reflected  in  Figure  24. 

As  indicated  in  this  figure,  at  all  levels  of  maintenance  trouble- 
shooting, the  FPTA  was  favored  as  an  aid  over  the  LTTA  by  the  non- 
experienced  Keesler  group  of  subjects.  The  opinions  expressed  for  FPTA 
usage  ranged  from  a value  of  62  to  125  with  a mean  value  of  110.6  mm. 

This  mean  value  would  represent  the  consensus  opinion  of  Keesler  subjects 
that  the  FPTA  would  be  considered  extremely  useful  in  this  type  of 
maintenance  activity.  The  LTTA  did  not  receive  this  high  an  acceptance. 
The  overall  values  for  LTTA  ranged  from  11  to  125  with  an  overall  value 
of  84.4  mm.  This  value  can  be  interpreted  as  exhibiting  the  consensus 
that  the  LTTA  would  be  of  some  help  but  is  not  favored  over  the  FPTA. 

On  other  questions,  the  Keesler  subjects  continued  to  indicate  a 
favoring  of  the  FPTA  over  the  LTTA.  The  response  favoring  use  of  the 
FPTA  for  some  level  of  maintenance  was  96  percent.  This  corresponded 
to  68  percent  for  the  LTTA.  Viewed  by  level  of  maintenance:  for  organi- 

zational level  troubleshooting,  all  were  of  the  opinion  that  FPTA  should 
be  used,  but  only  72  percent  favored  LTTA;  for  intermediate  level  the 
percentages  were  92  and  64.  At  the  organizational  level,  the  opinion 
favoring  FPTA  over  LTTA  was  statistically  significant  (tabular  x2  “ 

3.84,  calculated  x2  ” 5.98).  At  the  intermediate  level  no  statistically 
significant  difference  was  obtained  (calculated  x2  * 2.04). 

Military  Airlift  Command  Subjects 

The  effects  of  type  of  technical  documentation  on  troubleshooting 
performance  were  evaluated  by  means  of  an  analysis  of  variance  of  a 
three-factor  experiment  with  repeated  measures  on  the  second  and  third 
factors.  Unlike  the  Keesler  subjects,  the  MAC  subjects  were  exposed  to 
problems  using  all  three  documentation  types  (FPTA,  LTTA,  and  TO). 

As  with  the  Keesler  group,  three  separate  analyses  of  variance  were 
computed  for  the  MAC  data.  One  analysis  used  the  number  of  problems 
correctly  solved  as  the  criterion  measure.  The  second  analysis  used 
the  number  of  spare  parts  consumed  as  the  criterion.  The  third  analysis 
used  the  time  to  troubleshoot  and  repair  as  the  criterion.  The  results 
for  the  analysis  of  variance  for  number  of  problems  correctly  solved 
with  each  type  of  technical  documentation  are  presented  in  Table  5; 


How  do  you  feel  about  the  (FPTA,  LTTA)  as  an  aid  in  troubleshooting 
the  maintenance  problems  you  have  just  had  at  the  following  levels 
of  maintenance? 


Cockpit  Level 


LTTA  FPTA 
(92.1)  (109.4), 


Not 

Useful 
At  All 


Extremely 

Useful 


Bench  Level  to  Subassemblies 


LTTA 

(78.5) 


FPTA 

(111.8) 


Not 

Useful 
At  All 


Extremely 

Useful 


Subassemblies  to  Components 

LTTA 

I (82.6) 


FPTA 

(110.7), 


Not 

Useful 
At  All 


Extremely 

Useful 


Figure  24.  Opinion  Results  of  Keesler  Technical  Training  Center 
Subjects  to  Questions  about  Usefulness  of  FPTA  or 
LTTA  as  a Troubleshooting  Aid 
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rthe  analysis  of  variance  results  for  number  of  spare  parts  consumed 

in  Table  6;  and  the  analysis  of  variance  for  time  to  troubleshoot  and 
repair  in  Table  7. 

Following  the  procedure  used  at  Keesler,  an  attempt  was  made  to 
expose  subjects  to  either  organizational  or  maintenance  level  problems 
in  proportions  comparable  to  those  they  encountered  in  daily  operational 
maintenance  activities.  One  problem  was  added  to  the  sample  set  to 
allow  exposure  to  an  equal  number  of  problems  with  each  type  of  technical 
documentation.  Thus  each  subject  was  asked  to  solve  five  problems  with 
each  documentation  type.  For  the  first  analysis  (number  of  problems 
solved)  each  data  point  has  been  transformed  to  compensate  for  the 
unequal  weighting  of  number  of  problems  for  each  level  of  maintenance. 

The  transformation  entails  converting  the  performance  score  to  the 
proportion:  number  solved/number  presented.  For  computational  case, 

the  proportion  has  been  further  coded  by  omission  of  the  decimal  and 
division  by  10.  Thus,  subjects  could  attain  scores  ranging  from  0 to 
10  with  any  documentation  type  at  a specific  level  of  maintenance. 

Using  this  scoring  system,  a subject  correctly  solving  all  problems 
to  which  he  was  exposed  would  attain  a score  of  60.  With  the  MAC 
subjects,  a range  of  scores  from  36.6  to  60.0  was  obtained.  The  mean 
scores  for  each  documentation  type  by  level  of  maintenance  and  technician 
experience  level  were  presented  in  Figure  21. 

f The  summary  of  the  analysis  of  variance  depicted  in  Table  5 indicates 

that  four  of  the  effects  analyzed  were  statistically  significant.  Thus, 
the  effect  of  type  of  documentation  and  of  level  of  troubleshooting 
maintenance  on  performance  was  real.  In  addition,  two  interaction  effects 
were  present:  a strong  interaction  effect  existed  between  type  of 

documentation  being  used  and  the  level  of  troubleshooting  maintenance; 
a less  strong  but  real  interaction  effect  on  performance  also  existed 
between  experience  level  and  type  of  technical  documentation  being 
used. 

The  effect  of  level  of  maintenance  on  performance  is  graphically 
portrayed  in  Figure  B-l  (Appendix  B) . The  mean  proportion  of  problems 
solved  at  the  organizational  level  of  maintenance  was  0.98;  the  mean 
proportion  of  problems  for  the  intermediate  level,  0.85.  Since  only 
these  two  levels  of  maintenance  were  involved,  it  is  clear  that  perform- 
ance at  the  organizational  level  of  maintenance  was  significantly  better 
than  at  the  intermediate  level.  The  type  of  documentation  used  also 
significantly  affected  performance.  It  can  be  seen  from  Figure  B-2 
that  performance  was  highest  with  the  FPTA,  next  best  with  the  LTTA  and 
lowest  with  existing  Air  Force  TOs.  The  mean  proportion  of  number  of 
problems  correctly  solved  with  FPTAs,  LTTAs , and  TOs  were  0.97,  0.92, 
and  0.84,  respectively.  Use  of  the  Tukey  procedure  for  multiple 
comparisons  showed  that  no  significant  difference  in  performance  existed 
for  the  FPTA  and  the  LTTA,  but  that  performance  with  the  TO  was  signifi- 
cantly lower  than  for  either  the  FPTA  or  LTTA.  This  significance  was 
obtained  beyond  the  0.01  level. 
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The  interaction  effect  existing  between  type  of  documentation  and 
level  of  troubleshooting  maintenance  is  graphically  depicted  in  Figure 
B-3.  An  absence  of  interaction  would  be  denoted  by  the  presence  of 
parallel  lines.  It  can  be  seen  that  the  interaction  effect  is  caused 
primarily  by  the  influence  on  performance  of  the  TO,  which  differs  at  the 
two  levels  of  troubleshooting  maintenance.  Multiple  comparisons  by  means 
of  the  Tukey  procedure  showed  the  difference  in  means  for  FPTA  perform- 
ance was  not  real.  However,  the  difference  in  means  for  the  LTTA 
performance  achieved  significance  at  the  0.05  level,  and  the  difference 
for  TO  performance  was  significant  at  better  than  0.01.  Thus,  perform- 
ance is  generally  better  at  the  organizational  level  and  falls  off  at 
the  intt  mediate  level.  With  LTTAs,  performance  at  the  intermediate 
level  falls  off  faster  than  with  FPTA,  but  not  as  rapidly  as  with  TOs. 

The  second  interaction  effect  achieving  significance  in  the 
analysis  summarized  in  Table  5 was  tv,at  for  experience  level  by  type 
of  technical  documentation  being  useu.  This  interaction  is  portrayed 
graphically  in  Figure  B-4.  From  inspection  of  this  figure  it  would 
appear  that,  with  the  exception  of  the  use  of  TOs,  the  low  experience 
group  performed  better*'., than  did  the  more  experienced  group.  However, 
the  results  of  the  Tukey  procedure  do  not  support  this  contention. 

There  is  no  real  difference  between  the  two  experience  groups  for  per- 
formance with  the  FPTA  or  LTTA  documentation  types.  The  difference 
reflected  for  performance  with  TO  documentation  was  real  and  caused  the 
interaction  effect  of  documentation  type  and  experience  level  to  achieve 
significance. 

To  the  question  of  the  presence  of  an  interaction  effect  between 
type  of  technical  documentation  and  level  of  experience  for  number  of 
problems  correctly  solved,  the  data  again  dictate  a positive  response. 

The  degree  of  severity  of  this  interaction  is  reflected  in  Table  5. 

The  nature  and  direction  of  interaction  are  depicted  in  Figure  B-4. 

The  Tukey  procedure  identifies  the  interaction  existing  between  TOs 
and  experience  level  as  the  causal  factor  for  the  overall  interaction 
effect. 

, ' - 

Another  question  investigated  in  this  study  was  whether  performance 
with  the  three  types  of  technical  documentation  differed  for  organiza- 
tional and  intermediate  level  maintenance  troubleshooting  activities. 
Analysis  of  variance  results  (Table  7 ) established  the  presence  of  a 
strong  interaction  effect  for  technical  data  type  and  level  of 
maintenance.  Figure  20  provided  graphic  Insight  into  the  nature  of  this 
interaction.  Again,  the  effect  contributed  by  TOs  was  identified 
through  means  of  the  Tukey  procedure  as  being  the  causal  agent. 

The  analysis  of  variance  summary  for  number  of  spare  parts  used 
presented  in  Table  6 shows  two  main  effects  and  two  interaction  effects 
achieve  levels  of  statistical  significance  causing  the  hypothesis  of 
no  difference  to  be  rejected.  Type  of  technical  documentation  and 
level  of  maintenance  troubleshooting  are  the  significant  main  effects. 


81 


A significant  interaction  exists  for  experience  level  by  type  of 
technical  documentation  and  for  experience  level  by  type  of  technical 
documentation  by  maintenance  level.  Graphic  representation  of  these 
effects  is  presented  in  Figures  B-5,  B-6,  B-7,  and  B-8.  The  three-way 
interaction  for  experience  level,  type  of  technical  documentation,  and 
level  of  maintenance  troubleshooting  is  depicted  in  Figure  B-6.  Two 
factors  contributing  to  the  significant  interaction  are  quickly  discernible. 
First,  parts  usage  by  experience  level  reverses  for  LTTA  and  TO  when 
compared  against  FPTA.  With  FPTA  the  wore  experienced  used  more  parts. 
Second,  the  number  of  parts  used  by  level  of  maintenance  shows  a wider 
disparity  for  TOs  than  for  either  FPTAs  or  LTTAs.  The  effect  of  experi- 
ence is  most  constant  with  LTTAs. 

A summary  of  the  analysis  of  effects  of  each  of  the  document  types 
as  an  aid  to  troubleshooting  in  terms  of  time  to  locate  and  repair  a 
fault  is  presented  as  Table  7.  Inspection  of  this  table  shows  that 
maintenance  level  and  the  interaction  of  maintenance  level  and  technical 
data  were  highly  statistically  significant.  Further,  a trend  can  be  seen 
for  experience  level  to  interact  with  the  other  main  factors  of  the 
experiment:  technical  data  and  level  of  maintenance.  However,  though 

exhibiting  a strong  tendency  for  an  interaction  effect,  the  combination 
effect  of  any  other  factor  with  experience  level  was  not  strong  enough 
to  achieve  statistical  significance. 

The  two-way  interaction  between  experience  level  and  type  of 
technical  documentation  used  is  clearly  portrayed  in  Figure  B-8.  Again, 
the  reversal  of  number  of  parts  used  by  experienced  technicians  when 
using  FPTAs  is  clearly  evidenced. 

The  significant  difference  existing  among  types  of  technical  docu- 
mentation for  number  of  parts  used  in  troubleshooting  is  clearly  produced 
by  the  effect  of  TOs.  This  effect  is  displayed  in  Figure  B-7. 

The  significant  difference  existing  between  levels  of  maintenance 
troubleshooting  was  an  expected  difference  due  to  the  nature  of  the 
troubleshooting  tasks  involved  at  each  level  of  maintenance.  The 
potential  for  parts  replacement  is  much  greater  with  intermediate  mainte- 
nance level  troubleshooting  tasks.  This  relationship  of  parts  replaced 
to  level  of  maintenance  troubleshooting  is  depicted  in  Figure  B-5. 

In  this  study,  an  answer  was  sought  to  the  question  of  whether 
quality  of  performance  with  the  three  types  of  documentation  depends 
upon  the  experience  level  of  the  technician.  The  pattern  of  differences 
reflected  in  Figure  B-4  provides  evidence  for  an  answer  that,  with  the 
exception  of  TOs,  the  quality  of  performance  did  not  depend  upon  experi- 
ence level. 

Another  question  posed  was  whether  FPTA  performance  differed  from 
LTTA  performance,  and  if  both  FPTA  and  LTTA  performance  were  different 
from  TO  performance.  The  use  of  the  Tukey  procedure  on  the  data 


portrayed  in  Figure  B-2  provides  evidence  for  a positive  answer  to  the 
question  of  whether  or  not  performance  is  different  with  the  TO  than  it 
is  for  either  the  FPTA  or  LTTA.  No  real  difference  existed  for  perform- 
ance with  the  FPTA  and  LTTA. 

Summary  information  on  the  opinions  expressed  by  field-experienced 
technicians  toward  the  use  of  the  FPTA,  LTTA,  and  TO  is  contained  in 
Appendix  A to  this  volume.  Responses  to  an  opinion  question  concerning 
the  use  of  the  FPTA,  the  LTTA,  or  the  TO  for  intermediate  level  trouble- 
shooting of  equipment  were  fitted  along  a Likert-type  scale  125  mm  in 
length  (shown  in  Figure  25).  Possible  range  of  scores  was  from  0 (strong 
negative  valence  to  125  (strong  positive  valence) . For  this  question 
the  overall  responses  of  field  technicians  with  respect  to  FPTAs  ranged 
from  17  to  125,  with  a mean  response  of  92.7  mm.  The  overall  range  for 
LTTAs  was  0 to  125  with  a mean  response  of  81.6  mm.  For  TOs  the  opinion 
data  ranged  overall  from  9 to  117  with  a mean  response  of  58.0  mm.  The 
mean  score  values  expressed  by  the  MAC  subjects  about  the  usefulness  of 
the  FPTA,  LTTA,  and  TO  are  reflected  in  Figure  25.  Clearly,  for  this 
particular  maintenance  troubleshooting  activity,  field-experienced 
technicians  favored  either  FPTAs  or  LTTAs  over  the  present  Air  Force  TO. 

In  addition  to  the  technician's  opinion  about  the  use  of  FPTA,  LTTA 
or  TO  documentation,  each  technician  was  asked  to  give  his  opinion  with 
respect  to  the  ease  of  understanding  of  the  document  types.  Scores 
ranged  from  0 to  125  mm.  The  mean  response  for  each  type  of  documenta- 
tion was  as  follows:  FPTA — 89.3  mm,  LTTA — 76.0  mm,  TO — 43.3  mm.  The 
mean  responses  are  located  along  a scale  line  in  Figure  26. 
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How  do  you  feel  about  the  (FPTA,  LTTA,  TO)  as  an  aid  In  troubleshooting 
the  maintenance  problems  you  have  just  had  at  the  following  levels 
of  maintenance? 


Cockpit  Level 


TO  LTTA  FPTA 
(70.1)  (81.8)  (88.9) 


Not 

Useful 
At  All 


Bench  Level  to  Subassembly 


TO 

(59.2) 


LTTA 

(83.9) 


FPTA 

(95.3) 


Useful 
At  All 


Subassembly  to  Components 


TO 

(44.7) 


LTTA 

(79.2) 


FPTA 

(93.8) 


Not 

Useful 
At  All 


Figure  25.  Opinion  Results  of  MAC  Subjects  to  Questions  About  the 
Usefulness  of  FPTA,  LTTA  or  TO  as  a Troubleshooting  Aid 


FPTA 

(89.3) 


How  easy  was  it  to  understand  the  (FPTA/LTTA/TO)? 


Figure  26.  Mean  Response  for  Ease  of  Understanding  for  Each  Type 
of  Technical  Documentation  of  MAC  Subjects 
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Section  VII 


DISCUSSION 


The  results  of  the  experimental  evaluation  clearly  demonstrate  that 
the  use  of  the  proceduralized  troubleshooting  approach  led  to  signifi- 
cantly better  troubleshooting  than  the  use  of  the  TO.  This  finding  is 
consistent:  for  two  of  the  three  measures:  proportion  of  problems 

solved  and  spares  consumed.  The  only  exception  was  for  time  to  trouble- 
shoot at  the  organizational  level.  This  is  apparently  due  to  the  high 
degree  of  competence  of  the  technicians  who  can  frequently  recognize  the 
cause  of  the  trouble  after  only  a few  checks.  Time  to  troubleshoot 
was  significantly  greater,  however,  for  the  TO  for  intermediate  level 
troubleshooting. 

Comparisons  of  performance  using  the  FPTA  with  performance  using 
the  LTTA  indicate  that  use  of  the  FPTA  gives  superior  or  equal  performance 
on  all  measures.  The  FPTA  is  shown  to  be  superior  for  apprentice 
technicians  (no  experience)  on  all  three  measures.  For  the  experienced 
technicians,  FPTA  resulted  in  slightly  better  performance  on  all  measures 
except  time  to  troubleshoot  at  the  organizational  level.  Use  of  the  LTTA 
required  slightly  less  time  to  troubleshoot  at  the  organizational  level. 
Although  the  differences  in  performance  using  the  two  types  of  data  are 
not  greater  than  could  be  expected  by  chance,  the  fact  that  there  is  a 
consistent  trend  on  all  measures  does  support  the  observation  that  FPTAs 
lead  to  generally  superior  performance. 

A somewhat  unexpected  finding  was  the  essentially  equal  performance 
regardless  of  experience  level  of  the  technicians  for  troubleshooting  with 
FPTAs  at  both  levels  of  maintenance  troubleshooting.  A wider  discrepancy 
was  exhibited  for  performance  with  both  LTTA  and  TO  documentation  when 
comparing  organizational  level  against  intermediate  level  troubleshooting. 
Casual  observations  of  the  experimenters  during  the  running  of  this 
controlled  experiment  would  have  suggested  that  field-experienced 
technicians  rejected  the  FPTA  documentation  in  favor  of  the  LTTA  for 
the  reason  that  it  did  not  make  allowances  for  the  attained  skills  and 
knowledges  of  the  user.  This  contention  is  not  supported  by  the  data. 

Both  objective  performance  data  and  subjective  opinion  data  of  the 
technicians  provide  unqualified  support  for  the  use  of  the  FPTA  as  the 
most  favored  of  the  three  types  of  technical  documentation. 

One  of  the  anticipated  advantages  of  FPTAs  and  LTTAs  is  a reduction 
in  spare  parts  consumption.  The  results  of  the  experiment  confirm  this 
expectation.  Significantly  fewer  parts  were  used  with  both  FPTA  and 
LTTA  than  with  the  TO.  Nearly  twice  as  many  good  parts  were  unnecessarily 
replaced  when  the  TO  was  used  than  when  FPTAs  and  LTTAs  were  used.  These 
findings  suggest  that  significant  savings  in  spare  parts  could  result 
from  application  of  the  FPTA  and  LTTA  technology  to  operational  systems. 
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Another  important  observation  from  the  experimental  data  is  that  the 
performance  of  apprentice  technicians  using  FPTAs  and  LTTAs  approaches 
the  performance  of  both  categories  of  experienced  personnel.  Performance 
of  the  Keesler  subjects  with  FPTAs  was  essentially  the  same  as  the 
performance  of  the  experienced  technicians  with  FPTAs  and  better  than 
their  performance  with  the  TO.  The  performance  of  the  Keesler  subjects 
with  the  LTTA  approaches  the  performance  of  the  experienced  technicians 
with  the  LTTA  but  is  somewhat  lower.  However,  the  performance  of  the 
Keesler  subjects  with  the  LTTA  is  equal  to  the  performance  of  the  less 
than  six  month  group  with  TOs  and  approaches  the  performance  of  the  over 
six  month  group  with  the  TO.  The  significance  of  this  observation  is 
that  the  use  of  FPTAs  may  make  it  possible  for  new  technical  school 
graduates  to  become  effective  almost  immediately  upon  assignment  to  a 
maintenance  organization.  The  use  of  LTTAs  could  have  a similar  impact. 
However,  it  is  likely  that  more  OJT  would  be  required  before  performance 
could  reach  the  same  level.  This  benefit  could  have  a significant 
impact  on  the  operation  of  units  with  many  3-skill  level  personnel 
assigned. 

The  accuracy  analysis  indicates  that  there  is  relatively  little 
difference  in  the  technical  accuracy  and  completeness  of  the  data 
produced  under  the  rigid  specification  used  for  developing  the  FPTAs  and 
the  less  rigid  specification  used  for  developing  the  LTTAs.  Since  the 
procedures  specified  in  the  FPTA  specification  are  largely  required  to 
ensure  technical  accuracy,  one  might  be  tempted  to  conclude  that  the 
FPTA  specifications  are  not  valid.  However,  this  would  be  a hasty 
conclusion.  It  would  be  incorrect  to  conclude  that  the  tasks  specified 
in  the  FPTA  specification  are  not  necessary.  The  tasks  must  be  performed 
in  some  manner  to  develop  both  FPTAs  and  LTTAs.  The  question  is  whether 
the  tasks  should  be  accomplished  in  a specific  way  or  should  be  accomplished 
using  procedures  selected  by  the  contractor  and  approved  and  supervised 
by  the  procuring  agency.  The  same  basic  tasks  were  required  (by  the 
contract  but  not  by  MIL-M-38800A)  in  the  development  of  the  LTTA  prepared 
in  this  project  as  were  required  by  the  FPTA.  The  difference  was  that 
the  LTTA  contractor  was  given  freedom  in  determining  how  they  were  done 
and  the  procuring  agency  was  concerned  only  with  the  outcome. 

The  previous  analysis  would  suggest  that  an  appropriate  approach 
would  be  to  require  that  certain  tasks  be  performed  in  the  development 
of  the  aids  but  that  the  procedures  for  performing  them  not  be  rigidly 
specified.  The  required  procedures  would  require  a task  analysis 
(including  task  identification,  training/  technical  data  trade-off, 
action  tree  development,  etc.)  preparation  of  the  data  in  a specified 
format  and  validation.  This  approach  is  essentially  the  same  as  that 
used  in  developing  the  LTTAs  procured  in  this  study.  * 

Any  new  technical  order  has  many  errors.  The  FPTAs  and  LTTAs  do 
not  have  any  more  errors  than  would  be  expected  in  a new  technical  order. 
However,  the  nature  of  procedurallzed  data  requires  that  they  be  much 
more  accurate,  nearly  perfect.  This  is  due  to  the  fact  that  the 
technician  is  following  a predetermined  strategy  that  he  does  not 


87 


necessarily  understand  or  know  where  it  is  leading.  Thus,  if  a typo- 
graphical error  causes  him  to  go  to  page  16  instead  of  page  14,  he  would 
become  hopelessly  lost.  Time  and  financial  constraints  did  not  permit 
complete  verification  of  the  FPTAs  and  LTTAs.  As  a result  they  have  more 
errors  in  them  than  would  be  permissible  for  use  in  an  operational 
environment.  Further  verification  would  be  required  before  the  data 
could  be  used  operationally.  This  situation  provides  an  opportunity  to 
emphasize  that  a thorough  validation  and  verification  is  essential  for 
the  development  of  either  FPTAs  or  LTTAs.  It  is  strongly  recommended 
that  a thorough  validation  and  verification  be  required  for  all  FPTA  and 
LTTA  procurements. 

The  cost  comparison  clearly  indicates  that  the  FPTAs  are  more 
expensive  to  develop  than  LTTAs.  This  appears  to  be  due  largely  to  the 
fact  that  nearly  twice  as  many  pages  were  required  for  the  FPTAs  than 
for  the  LTTAs.  The  additional  pages  were  due  largely  to  the  requirements 
for  extensive  use  of  illustrations  and  the  increased  level  of  detail.  It 
is  believed  that  these  two  factors  account  for  a large  portion  of  the 
generally  better  performance  with  the  FPTA,  especially  with  the  inexperi- 
enced apprentice  technicians.  This  belief  is  based  on  observations  of 
technicians  using  the  LTTA.  The  most  apparent  causes  of  failure  were 
inability  to  locate  referenced  parts  and  insufficient  detail  in  the 
instructions  to  permit  the  less  experienced  technicians  to  complete  the 
step.  It  appears  then  that  any  decision  regarding  which  type  of  data 
to  procure  requires  a cost  trade-off  analysis.  The  trade-off  must  answer 
the  question  "Is  the  additional  benefit  worth  the  cost?"  The  answer 
depends  upon  several  factors  including:  the  skills  of  the  intended  user, 

the  expected  cost  savings  resulting  from  more  efficient  maintenance 
(reduced  maintenance  times  and  reduced  spares  consumption),  and  the 
impact  of  each  data  type  on  training. 

The  present  study  has  provided  valuable  information  relative  to  the 
comparative  costs  and  cost  benefits  of  FPTAs,  LTTAs,  and  TOs.  However, 
the  type  of  information  required  to  adequately  compare  costs  is  beyond 
the  scope  of  the  project.  It  has  not  adequately  compared  the  costs  of 
TOs  and  the  new  types  of  data.  To  satisfactorily  answer  this  question, 
it  would  be  necessary  to  obtain  specific  information  on  the  costs  to 
develop  FPTAs,  LTTAs,  and  TOs  on  the  same  (or  very  similar)  system  from 
the  same  data  base  (preferably  the  engineering  analysis)-  by  the  same  or 
very  similar  contractors  (in  terms  of  experience,  labor  rates,  etc.). 

This  type  of  data  would  provide  valid  input  for  a second  needed  study,  a 
life  cycle  cost  analysis.  The  life  cycle  cost  analysis  is  needed  to  evaluate 
the  Impact  of  FPTAs  and  LTTAs  on  maintenance  costs.  Such  a study  would 
provide  a means  of  estimating  the  cost  savings  due  to  reduced  spare  parts 
consumption  and  training  requirements. 

The  demonstration  that  apprentice  technicians  can  troubleshoot  at 
a level  of  effectiveness  equal  to  or  better  than  the  performance  of 
experienced  technicians  suggests  that  significant  personnel  and  training 
savings  could  result  from  the  application  of  the  FPTA  and  LTTA 
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technologies.  As  minimum,  apprentice  technicians  should  require  less  OJT 
and  be  effective  workers  earlier  in  their  careers.  The  performance  of 
the  apprentice  technicians  in  this  study  also  suggests  that  the  use  of 
FPTAs  and  LTTAs  could  make  it  feasible  to  reduce  the  length  of  Air  Training 
Command  resident  training.  For  example,  use  of  FPTAs  or  LTTAs  might  make 
it  possible  to  eliminate  sets  training  (training  on  representative  sets 
of  Air  Force  equipment)  from  the  resident  course.  This  could  signifi- 
cantly reduce  the  length  of  the  training  course  and  thereby  reduce 
training  costs.  Further  research  is  needed  to  study  and  evaluate  the 
potential  of  FPTAs  and  LTTAs  for  reducing  training. 


Section  VIII 


CONCLUSIONS  AND  RECOMMENDATIONS 


The  analysis  of  the  data  collected  in  the  study  indicates  that, 
given  technically  accurate  FPTAs  and  LTTAs  of  high  quality,  the  following 
conclusions  may  be  made: 

1.  Apprentice  electronics  technicians  with  no  field  experience  are 
able  to  troubleshoot  moderately  complex  electronic  systems 
effectively  using  FPTAs  and  LTTAs.  These  technicians  are  able 
to  troubleshoot  more  effectively  with  FPTAs  than  LTTAs. 

2.  Technicians  with  more  than  six  months  experience  on  moderately 
complex  electronic  systems  are  able  to  troubleshoot  more 
effectively  with  FPTAs  or  LTTAs  than  with  the  standard  TO  at 
the  intermediate  level  of  maintenance.  These  technicians  are 
able  to  troubleshoot  equally  well  with  FPTAs,  LTTAs,  and  TOs  at 
the  organizational  level  of  maintenance. 

3.  Technicians  with  some  experience  (but  less  than  six  months)  on 
moderately  complex  electronic  systems  are  able  to  troubleshoot 
more  effectively  using  FPTAs  or  LTTAs  than  with  the  standard 

TO  at  the  intermediate  level  of  maintenance.  These  technicians 
are  able  to  troubleshoot  equally  well  with  FPTAs,  LTTAs,  and 
TOs  at  the  organizational  level. 

4.  Apprentice  technicians  with  no  field  experience  using  FPTAs  are 
able  to  troubleshoot  as  effectively  at  the  intermediate  level 

as  experienced  technicians  (with  more  than  six  months  experience) 
using  standard  TOs.  The  apprentice  technicians  using  either 
FPTAs  or  LTTAs  are  able  to  troubleshoot  as  effectively  or  more 
effectively  at  the  intermediate  level  than  technicians  with 
some  experience  (less  than  six  months)  using  standard  TOs. 

5.  The  use  of  FPTAs  or  LTTAs  results  in  significantly  fewer  "good" 
parts  being  replaced  unnecessarily  than  when  standard  technical 
orders  are  used.  Use  of  FPTAs  or  LTTAs  at  the  intermediate 
maintenance  level  in  an  operational  environment  could  result  in 
significant  cost  savings  due  to  reduced  spare  parts  consumption. 

6.  For  experienced  technicians,  troubleshooting  at  the  organiza- 
tional level  is  performed  the  most  rapidly  when  the  TO  is  used. 
However,  at  the  intermediate • level,  troubleshooting  is  performed 
significantly  faster  when  either  the  FPTA  or  LTTA  is  used 
instead  of  the  standard  TO. 
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7.  Experienced  technicians  believe  that  the  TO  is  more  difficult 
to  understand  than  either  the  FPTA  or  LTTA. 

8.  Experienced  technicians  prefer  either  the  FPTA  or  LTTA  over  the 
standard  TO  for  use  in  troubleshooting  at  all  levels  of 
complexity  examined  (flight  line,  "black  box"  to  sub-asserably 
to  piece  part).  FPTA  is  preferred  over  LTTA. 

9.  There  is  insufficient  data  available  on  the  costs  of  FPTAs, 

LTTAs  and  TOs  to  permit  an  adequate  cost  comparison.  Further 
research  is  needed  to  determine  the  costs  of  producing  each 
type  of  data.  The  necessary  data  could  most  efficiently  be 
collected  when  processing  FPTAs  or  LTTAs  for  operational  use 
with  an  Air  Force  system. 

10.  Although  the  present  study  was  limited  to  electronic  maintenance, 
it  is  highly  probable  that  similar  results  would  be  obtained 
in  other  maintenance  areas.  However,  application  of  the 
technology  to  other  areas  is  necessary  before  a firm  conclusion 
can  be  made. 

As  a result  of  this  research  effort,  the  following  recommendations 
are  made: 


1.  Consideration  should  be  given  to  the  development  of  FPTAs  or 
LTTAs  for  any  future  technical  order  improvement  program.  It 
is  especially  important  that  these  types  of  data  be  considered 
for  intermediate  level  maintenance  where  the  benefits  are 
likely  to  be  the  greatest.  The  determination  of  which  type 

of  data  should  be  developed  should  be  based  on  an  analysis  of 
the  capabilities  of  the  intended  user,  type  and  complexity  of 
the  equipment,  and  the  maintenance  concept  for  the  system. 

2.  Proceduralized  troubleshooting  data  (FPTA,  LTTA,  or  hybrid) 
should  be  developed  for  one  or  more  sub-systems  of  a modern 
Air  Force  weapons  system.  Development  of  proceduralized 

data  for  such  a system  would  provide  the  opportunity  to  obtain 
data  on  the  costs  of  developing  the  data,  effectiveness  of  the 
data  in  an  operational  environment,  and  the  problems  associated 
with  the  use  of  the  data  (user  acceptance,  updating,  etc.).  It 
is  suggested  that  a sub-system  from  the  F-15  be  selected  as  the 
test  system.  An  F-15  system  would  provide  a test  of  the 
proceduralized  troubleshooting  technology  on  "state  of  the  art" 
equipment.  Also,  since  the  aircraft  is  operational,  it  would 
be  possible  to  collect  baseline  data  on  maintenance  of  the 
system  with  conventional  TOs.  If  this  recommendation  is  adopted, 
it  is  also  suggested  that  the  developer  be  required  to  maintain 
detailed  records  relative  to  the  cost  of  developing  the  data. 
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3.  Considering  the  importance  of  FPTAs  and  LTTAs  being  technically 
accurate,  it  is  essential  that  the  development  process  include 

a thorough  validation  and  verification.  It  is  strongly  recommended 
that  the  actual  equipment  be  provided  to  the  contractor  on  a 
dedicated  basis  for  use  in  the  task,  analysis  validation  and 
verification. 

4.  Additional  analyses  of  the  costs  and  cost  benefits  of  FPTAs  and 
LTTAs  should  be  made.  The  analyses  should  attempt  to  estimate 
the  potential  savings  in  spare  parts  and  training  resulting 
from  the  use  of  FPTAs  and  LTTAs. 

5.  Studies  should  be  conducted  to  determine  the  potential  impact 
of  FPTAs  and  LTTAs  on  the  training  and  personnel  systems  for 
reducing  the  life  cycle  costs  of  hardware  systems. 

6.  Strong  consideration  should  be  given  to  the  modification  of  the 
LTTA  format  to  include  more  of  the  procedural  material  found 

in  the  FPTA  (e.g.,  more  graphics  and  location  call-outs).  The 
resultant  documentation  format  would  capitalize  on  the  advantages 
observed  in  this  study  for  both  the  FPTA  and  the  LTTA. 
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APPENDIX  A 

(SUMMARY  DATA  AND  VERBATIM  COMMENTS  OF  KTTC 

AND  MAC  SUBJECTS  TO  QUESTIONS 
ABOUT  THE  FPTA,  LTTA,  AND  TO 


TABLE  A-l.  MEAN  RATINGS*  ASSIGNED  TO  FPTA  AND  LTTA  BY  KEESLER  SUBJECTS 
FOR  USE  AS  AN  AID  IN  TROUBLESHOOTING  AT  DIFFERENT  LEVELS  OF 
MAINTENANCE 


Cockpit 


Bench  Level 
to 

Subassembly 


Subassembly 

to 

Component 


Overall 

Rating 


FPTA  109. A 111.8  110.7  110.6 


LTTA  92.1  78.5  82.6  84. A 


* where  rating  assigned  can  range  from 
0-125,  with  125  the  highest  favorable 
rating 


TABLE  A-2.  MEAN  RATINGS*  ASSIGNED  TO  FPTA,  LTTA,  AND  TO  by  MAC 

SUBJECTS  FOR  USE  AS  AN  AID  IN  TROUBLESHOOTING  AT  DIFFERENT 
LEVELS  OF  MAINTENANCE 


Cockpit 

Bench  Level 
to 

Subassemb ly 

Subassembly 

to 

Component 

Overall 

Rating 

FPTA 

88.9 

95.3 

93.8 

92.7 

LTTA 

81.8 

83.9 

79.2 

81.6 

TO 

70.1 

59.2 

44.7 

58.0 

* where  rating  assigned  can  vary  from 
0 - 125,  with  125  the  highest  favorable 
rating 


TABLE  A-3.  RESPONSE  OF  KEESLER  SUBJECTS  TO  THE  QUESTION:  WOULD  YOU 

LIKE  TO  SEE  THE  AIR  FORCE  USE  FPTA  OR  LTTA  DOCUMENTATION 
FOR  ANY  OF  THE  FOLLOWING  LEVELS  OF  TROUBLESHOOTING 


FPTA 


LTTA 


Org. 

InC. 

Org. 

Int. 

YES 

25 

23 

18 

16 

NO 

0 

2 

7 

9 

TABLE  A-4.  RESPONSE  OF  MAC  TECHNICIANS*  TO  THE  QUESTION:  WOULD  YOU 

LIKE  TO  SEE  THE  AIR  FORCE  USE  FPTA  OR  LTTA  DOCUMENTATION 
FOR  ANY  OF  THE  FOLLOWING  LEVELS  OF  TROUBLESHOOTING 


FPTA 


LTTA 


Org. 

Int. 

Org. 

Int. 

YES 

32 

44 

31 

32 

NO 

14 

2 

15 

14 

* Responses  are  tabulated  for  all  46 
technicians  on  whom  data  was  collected. 

The  composition  of  the  46  technicians 
was  as  follows: 

used  as  subjects 
dropped  from  study 
extra  to  study 

radar  transition  technicians 


36 

1 

1 

8 


46 


TABLE  A-5.  MEAN  RATING  EACH  DOCUMENT  TYPE  ON  EASE  OF  UNDERSTANDING 
(WHERE  INDIVIDUAL  RATING  CAN  RANGE  FROM  0 - 125  AND  125 
INDICATES  GREATEST  EASE) 


Keesler 

MAC 

Subjects 

Subjects 

FPTA 

105.7 

89.6 

LTTA 

74.8 

77.0 

TO 

43.6 

100 


TABLE  A-6 


VERBATIM  COMMENTS  OF  SUBJECTS  TO  THE  QUESTION:  WHAT 

TROUBLESHOOTING  METHOD  WAS  THE  EASIEST  TO  FOLLOW?  WHY? 


MAC  SUBJECTS 


FPTA 


Because  everything  was  in  one  book  and  in  sequence  of  what  you 
would  want  next. 


They  took  you  step  by  step  through  the  troubleshooting,  which 
is  helpful  for  a beginner  on  the  system. 


They  had  best  steps.  At  bench  level  and  subassembly  checks 


Direct  referencing  to  the  particular  malfunction  was  stated 
at  each  step. 


Step  by  step  - geared  towards  each  and  every  operation 


Because  everything  has  been  taken  in  careful  account  for  the  tests 


Their  sequence  of  procedure  was  logical  and  easy  to  understand. 
They  provided  sufficient  information  to  locate  most  items  properly 
and  quickly  and  enabled  technician  to  find  the  checkout  procedure 
and  troubleshooting  steps. 


It  was  highly  specific  and  irritatingly  like  being  "programmed 
Still  effective  and  a well  written  text. 


Because  you  can  find  test  hook-ups  and  diagrams  on  the  pages 
you  are  working. 


It  was  easy  to  find  the  problem  but  would  take  longer  on  a real 
flight line. 


It  does  not  assume  that  the  person  reading  it  knows  anything  about 
the  system  under  check. 


Logical,  complete,  easy  to  find  everything  and  consistent 


Told  you  exactly  what  to  do  and  not  to  do.  Led  you  directly  to 
the  malfunction. 


All  controls,  test  points,  etc.  were  numbered,  making  items  easier 
to  locate.  Following  troubleshooting  procedures  was  just  a bit 
easier  than  LTTA's. 


TABLE  6.  Continued 


LTTA 

The  method  had  a logical  sequence  for  troubleshooting. 

Because  of  the  go-no-go  and  use  of  the  good  points  of  the  standard 
TO  such  as  schematic. 

It  provided  the  most  help  with  least  confusion. 

No  long  lengthy  reading  to  find  out  what  to  do  next  and  a person's 
skill  can  help  him  even  more. 

It  was  a logical,  computer-like  procedure. 

Logic  trees. 

The  block  diagrams  are  easier  to  use. 

LTTA’s  give  step  - step  sequence  - unable  to  get  lost. 

The  LTTA  gives  a "go-no-go"  type  of  procedure.  It  is  very  systematic 
and  gives  you  exact  instruction. 

Very  good  - the  middle  between  our  old  TOs  and  the  new  style  FPTA. 

The  best  of  both  - more  easily  accepted  into  Air  Force  society  now. 

Lean  more  towards  a knowledge  of  the  system  and  test  equipment  to 
work  properly. 

TO 

(No  selections  as  easiest  method) 

KEESLER  SUBJECTS 
FPTA 

More  detail. 

Explains  everything,  no  thought  involved. 

All  necessary  Information  was  there  in  the  book  as  to  test  equipment  i 

set  up  to  where  to  plug  in  wires. 

Because  it  told  you  more  of  what  you  have  to  do  and  look  for. 

Easier  to  understand 


102 


TABLE  6.  Concluded 
Keesler  Subjects  - FPTA 

More  Information,  procedure  is  "straight-forward";  well  defined. 

They  were  more  detailed  (the  steps)  which  made  it  hard  to  make  a 
mistake  in  a certain  procedure. 

Because  it  explains  things  simple  enough  that  an  inexperienced 
person  can  understand. 

Set  up  with  step  by  step  procedures  like  a report  outline. 

More  detailed  explanations.  Very  easy  to  understand  and  follow. 


\ 


ft 


It  tells  how  to  set  up  equipment  and  exactly  what  it  should  read. 
It  gives  detailed  information  for  finding  whatever  you  need  to 
find. 

All  the  diagrams  and  pictures  are  right  there  on  the  same  page  as 
the  procedures. 

More  diagrams. 

LTTA 

To  the  point  more. 


TABLE  A- 7 . TABULATION  OF  MOST  FREQUENT  RESPONSES  TO  THE  QUESTIONS 
INDICATED 


What  do  you  feel  are  problems  with  the  FPTA? 


Unpractical  for  organizational  level  5 
No  schematics  5 
Steps  too  lengthy  10 
Needs  further  refining  3 


What  do  you  feel  are  problems  with  the  LTTA? 


Have  to  use  TOs  with  them  4 
Hard  to  follow  17 
Unpractical  for  organizational  level  3 
Needs  more  diagrams  6 


What  do  you  feel  are  problems  with  the  TO? 


Need  better  troubleshooting  charts  8 
Hard  to  understand  26 
Diagrams  not  clear  3 
Requires  too  much  experience  8 


APPENDIX  B 


GRAPHIC  SUMMARIZATION  OF  STUDY  RESULTS 
FOR  MILITARY  AIRLIFT  COMMAND  SUBJECTS 


Proportion  of  Problems  Solved 


1.0 


Figure  B-2.  Mean  Proportion  of  Problems  Solved  for  Technical 
Documentation  Type,  MAC  Subjects 
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3.  Mean  Proportion  of  Problems  Solved  for  Type  of 
Technical  Documentation  by  Level  of  Trouble- 
shooting Maintenance,  MAC  Subjects 
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Figure  B-4 


Mean  Proportion  of  Problems  Solved  for  Technical 
Documentation  Type  by  Experience  Level  (in  months) 
MAC  Subjects. 


Figure  B-5.  Mean  Number  of  Parts  Used  for  Level  of  Maintenance 
Troubleshooting,  MAC  Subjects 


Hcan  Number  of  Parti  Used  for  Experience  Level  In 
Months  by  Type  of  Technical  Documentation  by  Level 
of  Maintenance  Troubleshooting,  MAC  Subjects. 
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APPENDIX  C 


DATA  COLLECTED  ON  A RESTRICTED  SAMPLE  OF  RADAR 
TECHNICIANS  TROUBLESHOOTING  INERTIAL  AND 
RADAR  NAVIGATION  SYSTEM  PROBLEMS 


BACKGROUND 

During  Che  period  of  time  that  the  field  evaluation  team  was 
running  subjects  at  McGuire  AFB  for  data  collection  in  the  main 
experiment,  data  were  also  collected  on  eight  technicians  in  AFS 
328X1,  Avionics  Navigation  Systems  Specialist.  These  technicians 
were  scheduled  to  participate  in  a test  program  conducted  by  the 
Military  Airlift  Command  to  study  the  feasibility  of  combining  the 
Computer,  Radar  and  Radio  shops.  Under  this  arrangement,  personnel 
in  AFS  328X1  would  be  required  to  work  on  the  AN/APN-147  and  AN/ASN-35. 
Normally  this  equipment  is  maintained  by  personnel  in  AFS  328X4. 

At  the  suggestion  of  McGuire  AFB  personnel,  the  328X1  technicians 
were  subjected  to  the  same  training  and  performance  testing  as  that 
completed  by  the  328X4  personnel  who  served  as  subjects  for  the  primary 
study.  This  provided  an  opportunity  to  collect  data  relative  to  the 
assertion  that  FPTAs  can  facilitate  transfer  of  technicians  to  systems 
for  which  they  have  not  had  specific  training  or  experience.  In 
addition,  it  provided  an  opportunity  for  the  AFS  328X1  technicians  to 
acquire  some  training  on  the  specialized  test  equipment  required  to 
maintain  the  AN/APN-147  and  AN/ASN-35  and  an  opportunity  to  gain  some 
experience  in  working  with  the  systems. 

It  is  recognized  that  the  small  sample  size  on  which  this  data 
were  collected  severely  restricts  the  usefulness  of  conclusions  drawn 
from  the  results  of  the  analysis.  Nevertheless,  it  is  believed  that 
the  obtained  data  can  provide  meaningful  Insights  for  future  actions, 
both  with  respect  to  continued  job  performance  aid  (JPA)  research,  and 
to  Air  Force  decisions  concerning  JPA  implementation.  For  these  reasons 
the  data  collected  on  the  radar  transition  group  are  presented  in  this 
appendix. 

SUBJECTS 

The  eight  subjects  in  the  sample  Included  both  active  and  reserve 
Air  Force  enlisted  technicians  who  hold  AFSC  328X1  (Avionics  Navigation 
Systems  Specialist).  These  eight  subjects  had  little  or  no  experience 
with  the  test  systems.  The  levels  of  experience  In  electronic  maintenance 
ranged  less  than  six  months  to  more  than  15  years. 


PROCEDURE 


The  data  collection  procedures  were  identical  to  those  utilized 
for  collecting  data  from  the  MAC  subjects  for  the  primary  study  (see 
page  57). 

RESULTS 

The  results  are  presented  in  two  formats.  They  are  presented  in 
the  form  of  a proportional  analysis  for  easy  interpretation.  This 
analysis  is  accompanied  by  a more  thorough  statistical  analysis. 

Proportional  Analysis 

The  proportion  of  problems  solved  using  each  type  of  data  is 
presented  in  Figure  C-l.  From  the  figure  it  may  be  observed  that  all 
problems  attempted  with  the  TO  as  an  aid  were  solved.  Performance 
with  the  FPTA  and  LTTA  was  somewhat  less  effective  with  90.9  and  88.9 
percent  of  the  problem  correctly  solved.  In  evaluating  this  observed 
difference,  one  should  keep  in  mind  that  the  number  of  problems 
attempted  with  each  type  of  data  is  very  small  (32  problems,  11  with 
FPTA,  9 with  LTTA,  and  12  with  the  TO).  Thus,  there  was  only  one 
problem  missed  with  the  FPTA  and  one  with  the  LTTA. 

At  the  Intermediate  level,  the  trend  was  reversed  with  use  of  the 
FPTA  and  LTTA  resulting  in  the  most  effective  performance.  Use  of  the 
FPTA  resulted  in  the  best  performance.  Performance  using  the  LTTA  was 
lower  than  for  the  FPTA,  but  higher  than  performance  with  the  TO. 

The  number  of  parts  used  by  level  of  maintenance  and  type  of 
technical  documentation  is  portrayed  in  Figure  C-2.  These  results 
should  be  interpreted  with  caution.  One  subject  was  responsible  for 
13  of  the  39  parts  used  by  the  group.  Eight  of  the  13  parts  were  used 
by  this  subject  in  solving  problems  st  the  intermediate  level  of 
maintenance  using  LTTA  documentation.  This  usage  rate  was  2.7  times 
higher  than  any  other  subject  and  has  an  inordinate  effect  on  the  results 
obtained. 

Statistical  Analysis 
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As  the  data  on  the  eight  radar  transition  subjects  were  collected 
within  a repeated  measures  design,  a total  of  47  degress  of  freedom 
was  available  for  analysis.  A stannary  table  of  the  results  of  an 
analysis  of  variance  for  number  of  problems  solved  is  presented  as 
Table  C-l.  This  analysis  revealed  that  a statistically  significant 
difference  existed  in  performance  for  the  two  levels  of  maintenance 
troubleshooting.  As  discussed  earlier,  this  difference  was  in  line 
with  expectations. 
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TABLE  C-l.  ANALYSIS  OF  VARIANCE  SUMMARY  TABLE,  RADAR 

TRANSITION  GROUP,  NUMBER  OF  PROBLEMS  SOLVED, 
TRANSFORMED  DATA 


df 

SS 

MS 

F 

P 

Subjects 

7 

29.66 

Maintenance  Level  (A) 

1 

79.57 

79.57 

17.08 

<0.01 

A « Subjects 

7 

32.65 

4.66 

Document  Type  (B) 

2 

48.95 

24.48 

3.43 

•05<P<. 10 

B x Subjects 

14 

99.91 

7.14 

Maintenance  Level  x Document 
Type  (A  x B) 

2 

190.53 

95.27 

31.97 

<0.001 

AB  x Subjects 

14 

41.67 

2.98 

TOTAL 

47 

522.94 

Performance  with  each  type  of  documentation,  graphically  portrayed 
in  Figure  C-3,  exhibited  a strong  trend  toward  a statistically  signifi- 
cant difference  (F  - 3.43)  but  did  not  achieve  the  F level  required  at 
the  0.05  level  of  significance  (3.74).  It  is  believed  that  a level 
required  for  statistical  significance  would  have,  been  achieved  with  a 
larger  simple  size. 

The  association  of  the  type  of  documentation  and  the  level  of 
maintenance  troubleshooting  produced  a highly  significant  interaction. 

The  effect  of  type  of  documentation  was  different  at  different  levels 
of  maintenance  troubleshooting.  This  association  is  graphically  portrayed 
in  Figure  C-4. 

The  analysis  of  variance  for  spare  parts  used,  summarized  in 
Table  C-2  led  to  inconclusive  results.  A trend  for  a real  difference 
in  number  of  spare  parts  used  by  level  of  maintenance  troubleshooting 
occurred.  The  disparate  opportunity  for  parts  replacement  in  the  two 
levels  of  maintenance  would  have  led  to  the  expectation  that  this 
difference  would  have  surpassed  the  level  required  for  statistical 
significance.  The  results  obtained  are  apparently  influenced  by  the 
atypical  performance  of  one  subject. 

DISCUSSION 

Although  the  data  reported  in  this  appendix  are  based  on  a sample 
which  is  too  small  to  justify  firm  conclusions,  it  does  have  significant 
implications  for  additional  research.  Proponents  of  procedurallzed 
troubleshooting  have  asserted  that  the  use  of  this  type  of  data  can 
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Figure  C-4.  Problem  Solving  Performance  (Transposed  Data)  for 
Che  Radar  Transition  Group  by  Level  of  Maintenance 
Troubleshooting  and  Type  of  Technical  Documentation 
(A  Score  of  80  Is  Possible) 
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TABLE  C-2.  ANALYSIS  OF  VARIANCE  SUMMARY  TABLE,  RADAR 

TRANSITION  GROUP,  NUMBER  OF  SPARE  PARTS  USED 
WITH  FPTA,  LTTA,  AND  TO 


df 

SS 

MS 

F 

P 

Subjects 

7 

17.48 

Maintenance  Level  (A) 

1 

13.01 

13.01 

4.12 

. 05<P<. 10 

A x Subjects 

7 

22.14 

3.16 

Document  Type  (B) 

2 

2.37 

1.18 

<1 

B x Subjects 

14 

16.96 

1.21 

Maintenance  Level  x Document 
Type  (A  x B) 

2 

3.30 

1.65 

<1 

AB  x Subjects 

14 

20.05 

1.43 

TOTAL 


47 


95.31 
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greatly  facilitate  the  transition  of  electronic  technicians  to  new 
systems.  The  observation  that  the  eight  technicians  with  no  experience 
or  training  on  the  test  systems  were  able  to  troubleshoot  as  well  or 
better  than  technicians  with  training  and  experience  on  the  systems 
using  FPTAs , LTTAs,  or  TOs  suggests  that  their  assertion  is  valid. 
However,  additional  research  is  needed  to  verify  these  observations. 


